Nov 01 22:20:30.052060 ip-10-0-141-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 22:20:30.052072 ip-10-0-141-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 22:20:30.052081 ip-10-0-141-223 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 22:20:30.052325 ip-10-0-141-223 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 22:20:40.129556 ip-10-0-141-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 22:20:40.129577 ip-10-0-141-223 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 28c7fdc202594a409cbbb05b671ff20b -- Nov 01 22:22:35.760904 ip-10-0-141-223 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:22:36.202497 ip-10-0-141-223 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:36.202497 ip-10-0-141-223 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:22:36.202497 ip-10-0-141-223 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:36.202497 ip-10-0-141-223 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208150 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208240 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208244 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208247 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208250 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.208244 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208253 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208257 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208261 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208264 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208266 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208270 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208272 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208275 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208277 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208280 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208283 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208285 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208288 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208291 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208293 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208296 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208299 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208301 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208304 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208307 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.208465 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208309 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208312 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208314 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208317 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208319 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208322 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208324 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208334 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208337 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208340 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208342 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208344 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208347 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208349 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208352 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208355 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208358 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208360 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208363 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208365 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.208953 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208368 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208370 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208373 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208375 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208378 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208381 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208383 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208386 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208388 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208391 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208394 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208397 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208399 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208401 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208404 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208407 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208410 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208412 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.209427 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208417 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208420 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208422 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208427 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208430 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208433 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208436 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208440 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208443 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208445 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208448 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208450 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208453 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208455 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208458 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208461 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208463 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208466 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208468 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.209885 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208472 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208475 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208479 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208483 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208486 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208489 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208492 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208494 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208497 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208500 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208502 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208505 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208508 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208511 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208513 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208516 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208518 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208521 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208524 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208526 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.210378 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208529 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208532 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.208535 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209331 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209337 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209340 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209343 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209346 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209349 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209352 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209354 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209357 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209360 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209363 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209367 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209372 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209375 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209379 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209381 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.210872 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209385 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209387 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209390 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209392 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209395 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209398 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209400 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209403 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209406 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209409 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209411 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209414 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209417 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209420 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209422 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209425 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209428 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209431 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209434 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.211332 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209436 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209439 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209442 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209445 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209447 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209450 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209452 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209455 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209458 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209460 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209463 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209465 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209468 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209470 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209473 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209475 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209478 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209480 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209482 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.211805 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209485 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209487 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209490 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209492 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209495 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209497 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209500 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209502 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209505 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209507 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209510 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209512 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209514 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209517 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209519 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209521 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209526 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209528 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.212267 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209531 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209533 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209536 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209539 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209542 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209544 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209547 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209550 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209552 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209554 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209556 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209559 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209561 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209564 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209566 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209571 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209575 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209579 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209582 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209584 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.212766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209587 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209589 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209592 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209595 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209597 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209620 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209624 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209628 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209633 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209636 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209638 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209640 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.209643 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210219 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210226 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210231 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210234 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210236 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.213251 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210242 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210245 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210247 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210250 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210252 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210255 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210257 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210260 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210263 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210266 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210268 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210271 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210274 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210276 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210279 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210281 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210284 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210287 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210289 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210292 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.213720 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210294 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210296 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210298 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210301 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210303 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210306 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210308 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210311 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210313 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210316 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210319 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210321 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210324 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210326 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210329 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210331 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210333 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210336 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210338 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210341 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.214192 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210343 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210346 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210349 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210351 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210354 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210356 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210359 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210362 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210364 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210367 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210369 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210372 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210374 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210376 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210379 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210382 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210384 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210387 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210389 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210392 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.214684 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210394 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210397 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210403 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210406 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210408 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210410 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210413 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210415 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210418 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210420 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210423 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210425 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210428 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210431 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210433 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210436 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210439 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210441 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210445 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.215160 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210449 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210452 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210455 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210458 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210460 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210463 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210465 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210468 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210471 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210473 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210476 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210478 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210480 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210485 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210489 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210493 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210497 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210500 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210502 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.215624 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210506 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.210509 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211617 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211630 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211637 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211643 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211653 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211656 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211660 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211665 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211668 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211671 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211674 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211678 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211681 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211684 2573 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211687 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211689 2573 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211692 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211695 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211700 2573 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211704 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211708 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211711 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:22:36.216083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211714 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211718 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211723 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211727 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211730 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211733 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211737 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211739 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211742 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211746 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211750 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211753 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211756 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211759 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211762 2573 flags.go:64] FLAG: --enable-server="true" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211764 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211769 2573 flags.go:64] FLAG: --event-burst="100" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211772 2573 flags.go:64] FLAG: --event-qps="50" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211775 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211778 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211780 2573 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211785 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211789 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211792 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211795 2573 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:22:36.216797 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211797 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211800 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211803 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211806 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211811 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211814 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211817 2573 flags.go:64] FLAG: --feature-gates="" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211821 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211824 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211828 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211831 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211834 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211838 2573 flags.go:64] FLAG: --help="false" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211841 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211845 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211848 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211851 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211855 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211859 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211862 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211865 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211867 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211870 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:22:36.217440 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211873 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211876 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211879 2573 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211882 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211885 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211888 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211890 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211893 2573 flags.go:64] FLAG: --lock-file="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211896 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211899 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211902 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211908 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211911 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211915 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211918 2573 flags.go:64] FLAG: --logging-format="text" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211921 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211924 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211927 2573 flags.go:64] FLAG: --manifest-url="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211930 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211934 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211937 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211941 2573 flags.go:64] FLAG: --max-pods="110" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211944 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211947 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211950 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:22:36.218009 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211953 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211956 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211959 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211962 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211969 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211972 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211975 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211978 2573 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211981 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211984 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211987 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211990 2573 flags.go:64] FLAG: --port="10250" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211993 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211996 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05fb184765a746cb2" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.211999 2573 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212002 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212005 2573 flags.go:64] FLAG: --register-node="true" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212007 2573 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212011 2573 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212014 2573 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212018 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212021 2573 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212024 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212027 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212030 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212033 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:22:36.218630 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212036 2573 flags.go:64] FLAG: --runonce="false" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212039 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212042 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212045 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212048 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212051 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212054 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212058 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212061 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212064 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212067 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212070 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212073 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212076 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212078 2573 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212081 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212087 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212089 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212092 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212097 2573 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212100 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212102 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212105 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212108 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212111 2573 flags.go:64] FLAG: --v="2" Nov 01 22:22:36.219512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212115 2573 flags.go:64] FLAG: --version="false" Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212120 2573 flags.go:64] FLAG: --vmodule="" Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212132 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212136 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212272 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212276 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212279 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212282 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212286 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212289 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212292 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212294 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212297 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212300 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212303 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212305 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212308 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212310 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212313 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212315 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.220581 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212318 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212321 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212324 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212329 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212332 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212335 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212338 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212340 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212343 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212345 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212348 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212350 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212353 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212357 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212359 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212362 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212364 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212367 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212369 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212372 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.221450 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212374 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212377 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212379 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212381 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212384 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212386 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212390 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212394 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212397 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212400 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212402 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212406 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212410 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212413 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212415 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212419 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212422 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212424 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212427 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.222335 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212429 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212432 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212434 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212437 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212439 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212442 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212445 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212448 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212451 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212453 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212455 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212458 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212460 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212463 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212466 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212468 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212471 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212473 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212475 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.223208 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212478 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212480 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212483 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212485 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212488 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212490 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212493 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212495 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212498 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212502 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212504 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212507 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212509 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212512 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212514 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212517 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212519 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212522 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212524 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.224084 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212528 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212530 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212533 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212535 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212538 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212540 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212543 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212545 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212548 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212550 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212552 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212555 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212557 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.212584 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212633 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212638 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.224580 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212641 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212644 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212647 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212650 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212652 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212655 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212659 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212661 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212664 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212666 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212669 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212671 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212674 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212676 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212678 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212681 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212683 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212687 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212689 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212692 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.225089 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212694 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212697 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212699 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212701 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212704 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212707 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212709 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212712 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212714 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212717 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212720 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212723 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212725 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212728 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212731 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212734 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212736 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212739 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212742 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212745 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.225569 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212747 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212750 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212752 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212755 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212757 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212759 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212762 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212764 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212766 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212770 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212773 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212776 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212779 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212781 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212784 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212786 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212789 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212791 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212794 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.226303 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212796 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212799 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212801 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212805 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212809 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212812 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212814 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212823 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212826 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212830 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212833 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212840 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212843 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212845 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212848 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212851 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212854 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212856 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212859 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.227120 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212861 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212864 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212866 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212870 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212872 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212875 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212877 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212880 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212882 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212884 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212887 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212890 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212892 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212894 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212897 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212899 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212902 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212904 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212906 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.227721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212909 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212911 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212914 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212916 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212920 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212923 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.212925 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.221805 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.221826 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.221848 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.221856 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221944 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221955 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221961 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221965 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221970 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221974 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221981 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.228349 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221986 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221990 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221994 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.221999 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222004 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222008 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222017 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222021 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222026 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222030 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222034 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222038 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222042 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222047 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222051 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222055 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222059 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222063 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222068 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.228944 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222073 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222077 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222089 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222094 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222098 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222103 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222107 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222112 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222116 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222120 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222124 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222129 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222133 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222137 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222141 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222145 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222150 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222154 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222158 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.229478 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222162 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222166 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222170 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222175 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222179 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222183 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222187 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222191 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222196 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222200 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222204 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222207 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222212 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222217 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222221 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222225 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222237 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222241 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222246 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222250 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.229958 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222254 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222259 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222263 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222267 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222272 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222276 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222280 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222284 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222288 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222297 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222301 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222305 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222309 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222313 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222317 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222321 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222325 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222329 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222333 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.230520 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222337 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222340 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222344 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222348 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222352 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222356 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222360 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222364 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222368 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222381 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222385 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222390 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222394 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222398 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222402 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222406 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222410 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222415 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222419 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.231069 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222423 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.222463 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222809 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222818 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222823 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222828 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222832 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222836 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222840 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222844 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222849 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222853 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222860 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222866 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222871 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.231796 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222876 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222881 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222886 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222890 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222894 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222898 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222903 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222907 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222922 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222926 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222931 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222935 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222939 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222942 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222947 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222951 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222955 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222959 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222964 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222968 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.232425 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222972 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222976 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222981 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222985 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222989 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222993 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.222997 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223001 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223005 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223009 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223014 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223018 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223022 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223026 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223030 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223035 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223039 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223043 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223048 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223052 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.233355 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223056 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223068 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223072 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223076 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223080 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223084 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223088 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223093 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223097 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223101 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223105 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223110 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223114 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223119 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223123 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223127 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223131 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223136 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223140 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.234016 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223144 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223149 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223153 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223159 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223174 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223179 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223184 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223188 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223192 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223196 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223201 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223204 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223209 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223213 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223216 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223227 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223232 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223236 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223240 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223244 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.234470 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223248 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223252 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223257 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223264 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223268 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223272 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223276 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223280 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223285 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223289 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223293 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223297 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223301 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223305 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.223347 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:36.235043 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223407 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223414 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223418 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223422 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223426 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223431 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223435 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223439 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223444 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223448 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223452 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223457 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223461 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223474 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223479 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223483 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223487 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223492 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223498 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:36.235422 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223504 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223508 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223516 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223520 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223524 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223529 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223533 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223538 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223542 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223546 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223550 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223555 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223559 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223563 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223567 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223571 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223576 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223580 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223584 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223588 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:36.236046 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223592 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223616 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223621 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223625 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223629 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223634 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223638 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223650 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223655 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223659 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223663 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223667 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223671 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223675 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223680 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223684 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223688 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223693 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223697 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:36.236760 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223701 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223707 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223712 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223718 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223723 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223729 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223734 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223738 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223743 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223746 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223751 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223755 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223759 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223763 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223767 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223771 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223775 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223780 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223784 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:36.237295 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223788 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223792 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223804 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223809 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223814 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223818 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223822 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223827 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223833 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223840 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223844 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223848 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223852 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223857 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223861 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223865 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223869 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223873 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223877 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223881 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:36.237779 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223885 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223889 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223895 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223899 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223903 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223908 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223911 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223915 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:36.223920 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.224011 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.228047 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.229252 2573 server.go:1014] "Starting client certificate rotation" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.229350 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.229389 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.232057 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:36.238245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.237051 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:36.250682 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.250666 2573 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:22:36.254577 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.254560 2573 log.go:25] "Validated CRI v1 image API" Nov 01 22:22:36.255501 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.255475 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:22:36.256902 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.256885 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:36.260570 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.260547 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 84325919-1e48-4972-9222-efa0f7d1624e:/dev/nvme0n1p3 93ef4de6-8423-4fe1-9302-603ee9e9d2dd:/dev/nvme0n1p4] Nov 01 22:22:36.260664 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.260568 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 22:22:36.265928 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.265804 2573 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:36.264260224 +0000 UTC m=+0.389284835 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095066 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e4ade4504eea73898a9accdf403d6 SystemUUID:ec2e4ade-4504-eea7-3898-a9accdf403d6 BootID:28c7fdc2-0259-4a40-9cbb-b05b671ff20b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:13:7f:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:13:7f:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:ce:e6:9c:c5:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:22:36.265928 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.265924 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:22:36.266109 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.266090 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:22:36.269553 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.269528 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:22:36.269717 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.269555 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-223.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:22:36.270435 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.270426 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:22:36.270465 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.270438 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:22:36.270495 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.270467 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:36.270495 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.270480 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:36.271758 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.271748 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:22:36.276377 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276365 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:22:36.276507 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276497 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:22:36.276537 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276530 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:22:36.276563 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276540 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:22:36.276563 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276551 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:22:36.276563 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276560 2573 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:22:36.276695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.276568 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:22:36.277068 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.277057 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:36.279476 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.279455 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:22:36.288299 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.288283 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:36.289168 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289156 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:22:36.289216 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289186 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:22:36.289346 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289336 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:22:36.289378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289352 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:22:36.289378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289359 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:22:36.289378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289364 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:22:36.289378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289371 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:22:36.289378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.289377 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:22:36.290198 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290189 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:22:36.290233 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290199 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:22:36.290233 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290208 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:22:36.290233 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290214 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:22:36.290233 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290222 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:22:36.290233 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290231 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:22:36.290979 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290970 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:22:36.290979 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.290979 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:22:36.295056 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.295032 2573 server.go:1263] "Started kubelet" Nov 01 22:22:36.295156 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.295090 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:22:36.295207 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.295115 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:22:36.295245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.295205 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:22:36.295862 ip-10-0-141-223 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:22:36.297804 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.296491 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:22:36.297804 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.297384 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:22:36.299199 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.299169 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-223.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 22:22:36.299957 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.299938 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hwgbh" Nov 01 22:22:36.300049 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.300024 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 22:22:36.300096 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.300024 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-223.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 22:22:36.302115 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.302094 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 22:22:36.303166 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.302059 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-223.us-east-2.compute.internal.18e40c02752327c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-223.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-223.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:36.295006147 +0000 UTC m=+0.420030758,LastTimestamp:2026-11-01 22:22:36.295006147 +0000 UTC m=+0.420030758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-223.us-east-2.compute.internal,}" Nov 01 22:22:36.304873 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.304856 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hwgbh" Nov 01 22:22:36.306525 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.306506 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:36.307901 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.307885 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:22:36.308017 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.308000 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.308738 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.308722 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:22:36.308908 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.308731 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:22:36.308908 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.308909 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:22:36.309070 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.308737 2573 factory.go:55] Registering systemd factory Nov 01 22:22:36.309070 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.308969 2573 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:22:36.309070 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309019 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:22:36.309070 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309026 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:22:36.309448 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309434 2573 factory.go:153] Registering CRI-O factory Nov 01 22:22:36.309448 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309447 2573 factory.go:223] Registration of the crio container factory successfully Nov 01 22:22:36.309558 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309489 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:22:36.309558 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309502 2573 factory.go:103] Registering Raw factory Nov 01 22:22:36.309558 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.309512 2573 manager.go:1199] Started watching for new ooms in manager Nov 01 22:22:36.310490 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.310474 2573 manager.go:319] Starting recovery of all containers Nov 01 22:22:36.318537 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.318511 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:36.322022 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.322008 2573 manager.go:324] Recovery completed Nov 01 22:22:36.322751 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.322729 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-223.us-east-2.compute.internal\" not found" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.325944 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.325930 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.327733 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.327716 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.327850 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.327745 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.327850 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.327756 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.328232 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.328219 2573 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:22:36.328232 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.328230 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:22:36.328310 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.328236 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.328310 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.328246 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:22:36.329096 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.329082 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.329133 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.329109 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.329133 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.329123 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.330337 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.330326 2573 policy_none.go:50] "Start" Nov 01 22:22:36.330378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.330342 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:22:36.330378 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.330353 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:22:36.332386 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.332376 2573 policy_none.go:44] "Start" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.378350 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.382489 2573 manager.go:341] "Starting Device Plugin manager" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.383133 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383145 2573 server.go:87] "Starting device plugin registration server" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383377 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383388 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383499 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383583 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.383591 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.384207 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.384246 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.390316 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.391559 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.391575 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.391593 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.391654 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 22:22:36.395011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.394812 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:36.483832 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.483794 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.484772 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.484752 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.484888 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.484783 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.484888 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.484795 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.484888 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.484821 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.492215 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.492196 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal"] Nov 01 22:22:36.492284 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.492263 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.493390 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.493377 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.493441 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.493397 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-223.us-east-2.compute.internal\": node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.493861 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.493847 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.493930 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.493877 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.493930 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.493892 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.496216 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.496202 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.496368 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.496354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.496412 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.496386 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.497017 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497003 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.497084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497024 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.497084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497035 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.497084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497048 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.497084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497071 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.497084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.497084 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.499247 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.499226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.499332 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.499251 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:36.500007 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.499993 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:36.500081 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.500024 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:36.500081 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.500059 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:36.533884 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.533868 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.535417 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.535399 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.537758 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.537742 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.610522 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.610483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.610522 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.610519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.610747 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.610544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.634061 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.634036 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.710793 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.710793 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.710906 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.710906 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710821 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.710906 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.710906 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.710862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.735013 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.734990 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.835695 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.835655 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:36.840832 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.840814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.843011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:36.842996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:36.936734 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:36.936699 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.037298 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.037222 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.137800 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.137766 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.229279 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.229248 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 22:22:37.229801 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.229392 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:37.229801 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.229449 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:37.238531 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.238517 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.308456 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.308358 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:37 +0000 UTC" deadline="2028-05-15 23:39:57.294654527 +0000 UTC" Nov 01 22:22:37.308456 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.308400 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13465h17m19.98625847s" Nov 01 22:22:37.338875 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.338835 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.439402 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.439375 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.539890 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.539857 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.627991 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:37.627954 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608dfbc50dc552b9f5f883d23954a6ba.slice/crio-625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d WatchSource:0}: Error finding container 625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d: Status 404 returned error can't find the container with id 625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d Nov 01 22:22:37.628371 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:37.628347 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b928c74164b2c9fe5433adca4981b0d.slice/crio-d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d WatchSource:0}: Error finding container d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d: Status 404 returned error can't find the container with id d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d Nov 01 22:22:37.632836 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.632819 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:22:37.640252 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:37.640233 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-223.us-east-2.compute.internal\" not found" Nov 01 22:22:37.665854 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:37.665830 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:39.037165 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.037146 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:39.051859 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.051828 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:39.056542 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.056519 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:22:39.066784 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.066767 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:39.222416 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.222330 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:39.606355 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.606279 2573 apiserver.go:52] "Watching apiserver" Nov 01 22:22:39.612978 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.612954 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:22:39.613303 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.613283 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-h74xz","kube-system/global-pull-secret-syncer-vhgqk","kube-system/konnectivity-agent-w5598","openshift-dns/node-resolver-t5wh4","openshift-image-registry/node-ca-xpgg7","openshift-multus/network-metrics-daemon-jhdh2","openshift-ovn-kubernetes/ovnkube-node-pj8xg","kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4","openshift-network-diagnostics/network-check-target-zvpsn","openshift-network-operator/iptables-alerter-54bgs","openshift-cluster-node-tuning-operator/tuned-ghlmj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal","openshift-multus/multus-7jrrm"] Nov 01 22:22:39.615823 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.615803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:39.615942 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.615896 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:39.617971 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.617945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.621085 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.620426 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.622094 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.621938 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:22:39.622094 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.621990 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.622094 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.622041 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.622310 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.622127 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:22:39.622874 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.622567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.623700 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.623685 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:22:39.623829 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.623812 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.623990 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.623833 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:22:39.624599 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.624563 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:22:39.624686 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.624643 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.624818 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.624802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.625677 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.625657 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.626004 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.625973 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.626079 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.626036 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:22:39.627076 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.627055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.627668 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.627651 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:22:39.627826 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.627812 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:22:39.627930 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.627899 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:22:39.629517 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.629298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.629517 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.629363 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:39.629799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.629776 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.630090 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.630076 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.630232 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.630217 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:22:39.630283 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.630257 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:22:39.632394 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.631843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.632394 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.631917 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:39.633843 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.633825 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.635058 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.635040 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:39.637871 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.637833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.639970 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.639943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.645114 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.645094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.646705 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.646799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646718 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.646799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.646799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.646799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.646799 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krx4b\" (UniqueName: \"kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.646985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zck\" (UniqueName: \"kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647235 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647207 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5wg\" (UniqueName: \"kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fdk\" (UniqueName: \"kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt58s\" (UniqueName: \"kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.647989 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dngg\" (UniqueName: \"kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.647976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56b7p\" (UniqueName: \"kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.648773 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648522 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648542 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5pv\" (UniqueName: \"kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.648997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qtk\" (UniqueName: \"kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.649562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.650261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.649366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.662615 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.662596 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5kfvm" Nov 01 22:22:39.669728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.669683 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5kfvm" Nov 01 22:22:39.720756 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.720732 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:39.722194 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722129 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:22:39.722307 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722197 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722609 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722750 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722770 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722771 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722884 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722953 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.722980 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723009 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723125 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723198 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.723259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723249 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:22:39.723748 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723453 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.723748 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723489 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.723748 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723536 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:22:39.723748 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.723714 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:22:39.725567 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.725523 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"608dfbc50dc552b9f5f883d23954a6ba","Type":"ContainerStarted","Data":"625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d"} Nov 01 22:22:39.726922 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.726898 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerStarted","Data":"d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d"} Nov 01 22:22:39.737784 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.737767 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:22:39.749977 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.749954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.750089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5pv\" (UniqueName: \"kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.750089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750227 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.750227 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.750227 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qtk\" (UniqueName: \"kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750360 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750750 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.750808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.750796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.751794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.751837 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.751867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.751842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.751964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752029 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752143 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752343 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krx4b\" (UniqueName: \"kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.756943 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752419 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.752686 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.752810 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:22:40.252759246 +0000 UTC m=+3.049534706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752882 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752896 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.752945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753021 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753057 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.757590 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753075 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753349 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zck\" (UniqueName: \"kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5wg\" (UniqueName: \"kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753408 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fdk\" (UniqueName: \"kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753713 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.758763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753731 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753901 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.753991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754020 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754086 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt58s\" (UniqueName: \"kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754429 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.759728 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.754989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755328 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755468 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755488 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.755530 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755558 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.755587 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:40.255570771 +0000 UTC m=+3.052346229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dngg\" (UniqueName: \"kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755675 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.760649 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755750 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755880 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755896 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.755997 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756066 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756363 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.761695 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756427 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756477 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b7p\" (UniqueName: \"kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.756786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757515 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757598 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757712 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.757833 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.758372 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:39.762645 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.758394 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:39.763332 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.758414 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:39.763332 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:39.758482 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:22:40.258458017 +0000 UTC m=+3.055233478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:39.763332 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.759679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.763332 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.763276 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.763518 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.763400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.763626 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.763600 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.763808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.763781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5pv\" (UniqueName: \"kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.764002 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.763785 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krx4b\" (UniqueName: \"kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.764384 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.764352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qtk\" (UniqueName: \"kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:39.764892 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.764871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zck\" (UniqueName: \"kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.766180 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.766158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5wg\" (UniqueName: \"kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.766526 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.766506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt58s\" (UniqueName: \"kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.766655 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.766637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dngg\" (UniqueName: \"kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.766878 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.766841 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fdk\" (UniqueName: \"kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:39.767293 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.767275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b7p\" (UniqueName: \"kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:39.857899 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.857797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.858046 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.857941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.859136 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.858976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.861185 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.861162 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:39.942484 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.942455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:22:39.945719 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.945276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7jrrm" Nov 01 22:22:39.950454 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.950437 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:39.955651 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.955617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:22:39.957076 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:39.957045 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc545d6bc_16aa_4c0d_b30e_2ac42f675347.slice/crio-2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5 WatchSource:0}: Error finding container 2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5: Status 404 returned error can't find the container with id 2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5 Nov 01 22:22:39.960689 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:39.960641 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5b9510_e846_4249_911e_6f04e319853f.slice/crio-96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87 WatchSource:0}: Error finding container 96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87: Status 404 returned error can't find the container with id 96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87 Nov 01 22:22:39.965619 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:39.965589 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2372e60_3b4e_407f_b0ed_6799496e9dd7.slice/crio-7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2 WatchSource:0}: Error finding container 7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2: Status 404 returned error can't find the container with id 7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2 Nov 01 22:22:39.976203 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.976181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:22:39.978330 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.978310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:22:39.981067 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.981046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:22:39.987309 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:39.987285 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb90f847_aeda_4739_a6ac_33626b1e17a8.slice/crio-ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757 WatchSource:0}: Error finding container ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757: Status 404 returned error can't find the container with id ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757 Nov 01 22:22:39.993766 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:39.993742 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41572fd2_3250_4c36_a7be_8c8f733c39c3.slice/crio-3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b WatchSource:0}: Error finding container 3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b: Status 404 returned error can't find the container with id 3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b Nov 01 22:22:39.999667 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:39.999647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:22:40.003113 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.003092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:40.006770 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.006725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:22:40.012147 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:40.012118 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f84be26_280f_4289_a270_45b31fcd6adc.slice/crio-8b25c1792cc5db0f65c601d58252d49933ce11ec528557d870e70d4a305690e8 WatchSource:0}: Error finding container 8b25c1792cc5db0f65c601d58252d49933ce11ec528557d870e70d4a305690e8: Status 404 returned error can't find the container with id 8b25c1792cc5db0f65c601d58252d49933ce11ec528557d870e70d4a305690e8 Nov 01 22:22:40.015553 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:40.015463 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bfdd4e_d511_4e4d_a225_17787716d664.slice/crio-74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160 WatchSource:0}: Error finding container 74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160: Status 404 returned error can't find the container with id 74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160 Nov 01 22:22:40.017935 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:22:40.017912 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b411a3_fb23_4551_894a_958b2404c226.slice/crio-7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9 WatchSource:0}: Error finding container 7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9: Status 404 returned error can't find the container with id 7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9 Nov 01 22:22:40.261326 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.261293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:40.261520 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.261353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:40.261520 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.261404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:40.261645 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261619 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:40.261704 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261646 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:40.261704 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261659 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:40.261792 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261722 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.261703178 +0000 UTC m=+4.058478639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:40.261950 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261912 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:40.262088 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.261965 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:40.262088 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.262004 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.261984437 +0000 UTC m=+4.058759924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:40.262088 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:40.262047 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.262019411 +0000 UTC m=+4.058794860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:40.674001 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.673901 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:39 +0000 UTC" deadline="2028-06-13 14:25:41.569604502 +0000 UTC" Nov 01 22:22:40.674001 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.673939 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14152h3m0.895670174s" Nov 01 22:22:40.738810 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.738697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w5598" event={"ID":"62bfdd4e-d511-4e4d-a225-17787716d664","Type":"ContainerStarted","Data":"74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160"} Nov 01 22:22:40.747128 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.747036 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" event={"ID":"5f84be26-280f-4289-a270-45b31fcd6adc","Type":"ContainerStarted","Data":"8b25c1792cc5db0f65c601d58252d49933ce11ec528557d870e70d4a305690e8"} Nov 01 22:22:40.751137 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.751053 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xpgg7" event={"ID":"41572fd2-3250-4c36-a7be-8c8f733c39c3","Type":"ContainerStarted","Data":"3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b"} Nov 01 22:22:40.754904 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.754784 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54bgs" event={"ID":"bb90f847-aeda-4739-a6ac-33626b1e17a8","Type":"ContainerStarted","Data":"ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757"} Nov 01 22:22:40.758211 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.758140 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerStarted","Data":"35e9a5ed2a4ec2855d5565ddfce2c6e93ca5ff316e67c7e68fc62000d6b249d8"} Nov 01 22:22:40.771686 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.771370 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5wh4" event={"ID":"d2372e60-3b4e-407f-b0ed-6799496e9dd7","Type":"ContainerStarted","Data":"7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2"} Nov 01 22:22:40.773777 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.773750 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7jrrm" event={"ID":"fb5b9510-e846-4249-911e-6f04e319853f","Type":"ContainerStarted","Data":"96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87"} Nov 01 22:22:40.778696 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.778644 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" event={"ID":"c545d6bc-16aa-4c0d-b30e-2ac42f675347","Type":"ContainerStarted","Data":"2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5"} Nov 01 22:22:40.783418 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:40.782991 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9"} Nov 01 22:22:41.269909 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.269539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:41.269909 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.269768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:41.269909 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.269801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270270 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270291 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270314 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270363 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:22:43.270347136 +0000 UTC m=+6.067122591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270699 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270738 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:43.270726325 +0000 UTC m=+6.067501780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270777 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.270835 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.270798 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:22:43.270790537 +0000 UTC m=+6.067565979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.675026 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.674958 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:39 +0000 UTC" deadline="2028-06-16 01:10:23.158001944 +0000 UTC" Nov 01 22:22:41.675026 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.675023 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14210h47m41.482982846s" Nov 01 22:22:41.722900 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.722869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:41.723085 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.723012 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:41.723348 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.723326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:41.723429 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.723355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:41.723472 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.723442 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:41.723551 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:41.723529 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:41.799215 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.799175 2573 generic.go:356] "Generic (PLEG): container finished" podID="5b928c74164b2c9fe5433adca4981b0d" containerID="97af0e3a36d0578cb05642ecc0ee672dda39c212771385ae162b390c6819f716" exitCode=0 Nov 01 22:22:41.799379 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:41.799237 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerDied","Data":"97af0e3a36d0578cb05642ecc0ee672dda39c212771385ae162b390c6819f716"} Nov 01 22:22:42.842274 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:42.842235 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerStarted","Data":"b5c607e8a20a65dfab367f5a131a061ce50bea51627393a73f012caba26d54ab"} Nov 01 22:22:42.859299 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:42.859232 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" podStartSLOduration=3.859215579 podStartE2EDuration="3.859215579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:42.857833178 +0000 UTC m=+5.654608665" watchObservedRunningTime="2026-11-01 22:22:42.859215579 +0000 UTC m=+5.655991044" Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.289385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.289458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.289497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.289652 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.289673 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.289685 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.289745 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.289727823 +0000 UTC m=+10.086503281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.289985 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.290029 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.290016162 +0000 UTC m=+10.086791611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.290081 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:43.290139 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.290111 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.290100727 +0000 UTC m=+10.086876175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:43.722117 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.722083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:43.722291 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.722221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:43.722632 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.722613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:43.722744 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.722722 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:43.722812 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.722774 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:43.722891 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:43.722840 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:43.852477 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.852437 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"608dfbc50dc552b9f5f883d23954a6ba","Type":"ContainerStarted","Data":"6e30a9906548e531ba31fc8246b9796408bec177e0ef6fe71d900c255dfcb057"} Nov 01 22:22:43.857431 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.857396 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="35d300d0f77ddd82999994ee018402d0a4e86a21f5dfa202410bbf1aec63edb3" exitCode=0 Nov 01 22:22:43.858575 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.858549 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"35d300d0f77ddd82999994ee018402d0a4e86a21f5dfa202410bbf1aec63edb3"} Nov 01 22:22:43.896435 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:43.896358 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" podStartSLOduration=4.896339731 podStartE2EDuration="4.896339731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:43.869184095 +0000 UTC m=+6.665959561" watchObservedRunningTime="2026-11-01 22:22:43.896339731 +0000 UTC m=+6.693115197" Nov 01 22:22:44.868011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:44.867899 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w5598" event={"ID":"62bfdd4e-d511-4e4d-a225-17787716d664","Type":"ContainerStarted","Data":"84b4ef35fc6741c1c08dd2bd70f84115f5354bf85147dcfbc9e3f6a9bf8b67f0"} Nov 01 22:22:45.722928 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:45.722893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:45.723129 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:45.723041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:45.723462 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:45.723440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:45.723573 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:45.723533 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:45.723637 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:45.723602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:45.723733 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:45.723687 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:45.872337 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:45.872304 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xpgg7" event={"ID":"41572fd2-3250-4c36-a7be-8c8f733c39c3","Type":"ContainerStarted","Data":"a303307b9b5cce4c6113cceb3eefc0135147c0b6b434a50fe5effe588c23c985"} Nov 01 22:22:45.886361 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:45.886305 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w5598" podStartSLOduration=3.910929896 podStartE2EDuration="7.886287652s" totalImagesPullingTime="3.975357756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:44.887038435 +0000 UTC m=+7.683813898" watchObservedRunningTime="2026-11-01 22:22:45.886287652 +0000 UTC m=+8.683063116" Nov 01 22:22:46.242401 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:46.242251 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:46.242815 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:46.242755 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:46.257184 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:46.257128 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xpgg7" podStartSLOduration=3.321254788 podStartE2EDuration="8.257112793s" totalImagesPullingTime="4.935858005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:45.887571007 +0000 UTC m=+8.684346471" watchObservedRunningTime="2026-11-01 22:22:46.257112793 +0000 UTC m=+9.053888257" Nov 01 22:22:46.882994 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:46.882955 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" event={"ID":"c545d6bc-16aa-4c0d-b30e-2ac42f675347","Type":"ContainerStarted","Data":"04b20040243d8ec88ca7d73abcfbba9f022179f9494e177ebdcfa20eb1406cff"} Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.318183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.318250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.318287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.318428 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.318447 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.318460 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.318520 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:22:55.318501964 +0000 UTC m=+18.115277410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.318966 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.319014 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:55.319000024 +0000 UTC m=+18.115775473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.319070 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:47.319143 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.319106 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:22:55.319092815 +0000 UTC m=+18.115868259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:47.620915 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.620883 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:22:47.715677 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.715572 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:22:47.620911769Z","UUID":"3a2144ab-0a29-4d06-83e9-f1b639485e33","Handler":null,"Name":"","Endpoint":""} Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.723177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.723309 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.723397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.723504 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.723576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:47.723674 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.723724 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:22:47.724141 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.723764 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:22:47.887087 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.887040 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5wh4" event={"ID":"d2372e60-3b4e-407f-b0ed-6799496e9dd7","Type":"ContainerStarted","Data":"a042e13e14ceca4962fc7add871ba12210b23bcaa8147cb03080aa7ac4b077aa"} Nov 01 22:22:47.889800 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.889735 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7jrrm" event={"ID":"fb5b9510-e846-4249-911e-6f04e319853f","Type":"ContainerStarted","Data":"fd29276c2840e565272808610e97fc6ab06010a0c2ad3923651e120a1755b32a"} Nov 01 22:22:47.894675 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.894654 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:47.894840 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.894815 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" event={"ID":"c545d6bc-16aa-4c0d-b30e-2ac42f675347","Type":"ContainerStarted","Data":"e43cf455fd1d1c8291737ec03773338b40c2b345f2eb6af8e7f902176ac218e5"} Nov 01 22:22:47.924275 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.924225 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t5wh4" podStartSLOduration=3.207102105 podStartE2EDuration="10.924211148s" totalImagesPullingTime="7.717109043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:47.904778205 +0000 UTC m=+10.701553671" watchObservedRunningTime="2026-11-01 22:22:47.924211148 +0000 UTC m=+10.720986612" Nov 01 22:22:47.924788 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:47.924751 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7jrrm" podStartSLOduration=3.095547955 podStartE2EDuration="10.924744883s" totalImagesPullingTime="7.829196928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:47.923944379 +0000 UTC m=+10.720719844" watchObservedRunningTime="2026-11-01 22:22:47.924744883 +0000 UTC m=+10.721520347" Nov 01 22:22:48.899621 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:48.899315 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" event={"ID":"c545d6bc-16aa-4c0d-b30e-2ac42f675347","Type":"ContainerStarted","Data":"dd113624f51bd9d7b07573ae5397ac19610ed7ecfb51d3a5f0e29d5b1edf6741"} Nov 01 22:22:48.901617 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:48.901586 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="ac41287c52579b7b41b1bf0016036735a990815e30f39de075ccc907f53ed842" exitCode=0 Nov 01 22:22:48.901739 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:48.901675 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"ac41287c52579b7b41b1bf0016036735a990815e30f39de075ccc907f53ed842"} Nov 01 22:22:48.959065 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:48.959009 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" podStartSLOduration=3.666052646 podStartE2EDuration="11.958988331s" totalImagesPullingTime="8.292935685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.917149528 +0000 UTC m=+11.713924994" watchObservedRunningTime="2026-11-01 22:22:48.958988331 +0000 UTC m=+11.755763795" Nov 01 22:22:49.720339 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.720308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:49.720339 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.720339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:49.720512 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:49.720445 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:49.720572 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.720534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:49.720661 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:49.720637 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:49.720755 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:49.720735 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:49.905185 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.905149 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" event={"ID":"5f84be26-280f-4289-a270-45b31fcd6adc","Type":"ContainerStarted","Data":"6344b53bdfd44009506398089c697bcf8eb2e88c05620b165d87ecebcf531f0e"} Nov 01 22:22:49.906557 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.906531 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54bgs" event={"ID":"bb90f847-aeda-4739-a6ac-33626b1e17a8","Type":"ContainerStarted","Data":"feb409ed198beb2bc3843c56dfb1b14a4e340122a45af07bb9528e83e88b2b88"} Nov 01 22:22:49.908545 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.908524 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="e9a5469b517408c71dfb019e66f6e73fa9584bc400655d90083378b3273c21b4" exitCode=0 Nov 01 22:22:49.908658 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.908570 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"e9a5469b517408c71dfb019e66f6e73fa9584bc400655d90083378b3273c21b4"} Nov 01 22:22:49.921777 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.921740 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" podStartSLOduration=2.206753329 podStartE2EDuration="11.921728871s" totalImagesPullingTime="9.714975542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:49.921523531 +0000 UTC m=+12.718298997" watchObservedRunningTime="2026-11-01 22:22:49.921728871 +0000 UTC m=+12.718504334" Nov 01 22:22:49.935902 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:49.935846 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-54bgs" podStartSLOduration=3.912745864 podStartE2EDuration="11.935833392s" totalImagesPullingTime="8.023087528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:49.935202551 +0000 UTC m=+12.731978015" watchObservedRunningTime="2026-11-01 22:22:49.935833392 +0000 UTC m=+12.732608855" Nov 01 22:22:50.428479 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:50.428441 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:50.428646 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:50.428598 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:50.429088 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:50.429063 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:22:50.914812 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:50.914779 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="dbb0f0b86811ed59edfacc6d2d25c8c4633ada32570db890f700ebdafff41b89" exitCode=0 Nov 01 22:22:50.915431 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:50.914906 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"dbb0f0b86811ed59edfacc6d2d25c8c4633ada32570db890f700ebdafff41b89"} Nov 01 22:22:51.720124 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:51.720087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:51.720299 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:51.720228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:51.720786 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:51.720760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:51.720952 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:51.720907 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:51.721024 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:51.720953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:51.721085 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:51.721063 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:52.922737 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:52.922697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerStarted","Data":"770f613f98c7bd252bd8aebd239bce12f1bc57f98d2753d72b1dc2a4b4d90d65"} Nov 01 22:22:53.720411 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:53.720218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:53.720572 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:53.720290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:53.720572 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:53.720510 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:53.720572 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:53.720313 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:53.720727 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:53.720572 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:53.720727 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:53.720635 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:53.930566 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:53.929092 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="770f613f98c7bd252bd8aebd239bce12f1bc57f98d2753d72b1dc2a4b4d90d65" exitCode=0 Nov 01 22:22:53.930566 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:53.929154 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"770f613f98c7bd252bd8aebd239bce12f1bc57f98d2753d72b1dc2a4b4d90d65"} Nov 01 22:22:54.934642 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:54.934603 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="5e2c64c6620c7090af1e90024a4cac01d2371710f542b755fb2d5e987d1d824f" exitCode=0 Nov 01 22:22:54.935075 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:54.934655 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerDied","Data":"5e2c64c6620c7090af1e90024a4cac01d2371710f542b755fb2d5e987d1d824f"} Nov 01 22:22:55.380280 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.380183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:55.380280 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.380255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.380296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380310 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380386 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380409 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380427 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380391 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.380369703 +0000 UTC m=+34.177145158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380441 2573 projected.go:196] Error preparing data for projected volume kube-api-access-4jpfb for pod openshift-network-diagnostics/network-check-target-zvpsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380461 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.380442217 +0000 UTC m=+34.177217664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:55.380521 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.380483 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb podName:d5ca1661-6040-419a-9037-1241908c41fa nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.380471794 +0000 UTC m=+34.177247241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4jpfb" (UniqueName: "kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb") pod "network-check-target-zvpsn" (UID: "d5ca1661-6040-419a-9037-1241908c41fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:55.720457 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.720422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:55.720640 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.720489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:55.720640 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.720493 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:55.720640 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.720574 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:55.720759 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.720705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:55.720834 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:55.720815 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:55.940939 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.940902 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h74xz" event={"ID":"d3183d19-6697-470b-b5ff-cad0136248fa","Type":"ContainerStarted","Data":"f2fe2448894882a634bb612c992255f42146d5b8b9175d1330a674bebd913b29"} Nov 01 22:22:55.967659 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:55.967610 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h74xz" podStartSLOduration=8.194727853 podStartE2EDuration="18.967597433s" totalImagesPullingTime="10.77286958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:55.967291895 +0000 UTC m=+18.764067371" watchObservedRunningTime="2026-11-01 22:22:55.967597433 +0000 UTC m=+18.764372897" Nov 01 22:22:57.721466 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.721272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:57.721829 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:57.721505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:57.721829 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.721393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:57.721829 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:57.721569 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:57.721829 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.721354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:57.721829 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:57.721618 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:57.948631 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948598 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"7f768512590931b2de6a4d52e77f1767a4ee11028000de0f0c407c35011b1bd8"} Nov 01 22:22:57.948796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948667 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"de161fd7b9bff916d8dd898a68c7ce35277821e8e2c993f4a52c63e13e95f520"} Nov 01 22:22:57.948796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948696 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"88caf2bc4a574c392cc762f6846cfdb35c01f260ad4da46ee7a67fbb569b2914"} Nov 01 22:22:57.948796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948725 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"74c7d442be64bc2a06b63082d42a0d878f40d8d6b2655a4fe3f2c607e3850c46"} Nov 01 22:22:57.948796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948752 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"3e17bb04a65b7c22a4803fe77e2b09bd450a789764dfb65c95df30b2b8a53158"} Nov 01 22:22:57.948796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:57.948779 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"d56d8132f2f06954b74b62dced86194a4f231410ebe90ac8e1aa9a50da06b03b"} Nov 01 22:22:59.720811 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:59.720770 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:22:59.721411 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:59.720780 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:22:59.721411 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:59.720917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:22:59.721411 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:59.720933 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:22:59.721411 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:59.721036 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:22:59.721411 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:22:59.721139 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:22:59.956161 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:22:59.956121 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"b4250c422e434a4f3bff3b866c13ca981602372e4bc09dbe1cc0289178b846cc"} Nov 01 22:23:01.720115 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:01.720083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:01.720115 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:01.720113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:01.720620 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:01.720091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:01.720620 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:01.720182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:23:01.720620 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:01.720250 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:23:01.720620 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:01.720354 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:23:02.968346 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.968310 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" event={"ID":"81b411a3-fb23-4551-894a-958b2404c226","Type":"ContainerStarted","Data":"e7db011fbc09055e5961a40adae24efc05e323094443a95458d28198152b278a"} Nov 01 22:23:02.969058 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.968471 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:02.969058 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.968507 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:02.969058 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.968540 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:02.985120 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.985095 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:02.987348 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.987329 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:02.999514 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:02.999478 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" podStartSLOduration=7.549006595 podStartE2EDuration="24.999468294s" totalImagesPullingTime="17.450461699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.998993754 +0000 UTC m=+25.795769267" watchObservedRunningTime="2026-11-01 22:23:02.999468294 +0000 UTC m=+25.796243757" Nov 01 22:23:03.702137 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.702105 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvpsn"] Nov 01 22:23:03.702318 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.702274 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:03.702409 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:03.702387 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:23:03.703011 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.702985 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhgqk"] Nov 01 22:23:03.703119 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.703096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:03.703212 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:03.703191 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:23:03.713360 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.713338 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhdh2"] Nov 01 22:23:03.713477 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:03.713439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:03.713548 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:03.713528 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:23:05.720133 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:05.720100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:05.720133 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:05.720127 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:05.720562 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:05.720103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:05.720562 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:05.720231 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:23:05.720562 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:05.720341 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:23:05.720562 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:05.720396 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:23:07.721198 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:07.720989 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:07.721613 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:07.721034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:07.721613 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:07.721276 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:23:07.721613 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:07.721067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:07.721613 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:07.721342 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:23:07.721613 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:07.721470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:23:09.721227 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:09.721177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:09.721712 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:09.721371 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvpsn" podUID="d5ca1661-6040-419a-9037-1241908c41fa" Nov 01 22:23:09.721808 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:09.721199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:09.722120 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:09.722071 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhdh2" podUID="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" Nov 01 22:23:09.722259 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:09.721818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:09.722480 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:09.722461 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhgqk" podUID="9445ebf3-feba-48ea-8eb6-882fd83343ab" Nov 01 22:23:10.555085 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.555055 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeReady" Nov 01 22:23:10.555242 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.555171 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 22:23:10.589743 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.589714 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5j2lt"] Nov 01 22:23:10.623626 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.623592 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jpxjw"] Nov 01 22:23:10.623814 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.623744 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:10.634982 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.634438 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.639037 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.639002 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtzbw"] Nov 01 22:23:10.639121 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.639083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:10.653413 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.653379 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:23:10.653609 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.653587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:10.667281 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.667257 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz"] Nov 01 22:23:10.667388 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.667361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.671720 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.671700 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kgdt\"" Nov 01 22:23:10.671720 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.671700 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 22:23:10.671920 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.671782 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 22:23:10.671920 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.671835 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 22:23:10.691523 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.691500 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2rlrx"] Nov 01 22:23:10.691636 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.691597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.704448 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.704429 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7"] Nov 01 22:23:10.704579 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.704563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.723083 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.723063 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nkrwh"] Nov 01 22:23:10.723369 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.723187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.734689 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.734664 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 22:23:10.734779 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.734692 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.735019 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.735003 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.735089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.735021 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.735089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.735024 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 22:23:10.735089 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.735067 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.735204 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.735163 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.736407 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.736392 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 22:23:10.737858 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.737832 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 22:23:10.747570 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.747553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.747570 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.747564 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nbf7s"] Nov 01 22:23:10.768805 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.768784 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls"] Nov 01 22:23:10.768914 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.768900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.771387 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.771368 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 22:23:10.771481 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.771425 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 22:23:10.771481 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.771449 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 22:23:10.771481 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.771449 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 22:23:10.772084 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.772071 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 22:23:10.783270 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.783348 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.783417 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783456 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrqx\" (UniqueName: \"kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.783485 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpx7\" (UniqueName: \"kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7\") pod \"kas-connection-checker-685895c7c9-jpxjw\" (UID: \"06fbbe23-1f70-4c1e-b82f-5fa72a4166db\") " pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:10.783485 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783543 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p689n\" (UniqueName: \"kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n\") pod \"kas-connection-checker-685895c7c9-5j2lt\" (UID: \"65225fe4-74e2-4b06-9169-7dce163b2fec\") " pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:10.783597 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.783654 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783654 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.783734 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783765 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783798 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69zf\" (UniqueName: \"kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf\") pod \"kas-connection-checker-685895c7c9-wtzbw\" (UID: \"29c2c4f7-79ec-444c-be78-ce43edf6850c\") " pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:10.783798 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783927 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.783927 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.783838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.793254 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.793233 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:23:10.793372 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.793358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.812056 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.812007 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx"] Nov 01 22:23:10.812159 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.812125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.835255 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835229 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 22:23:10.835359 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835284 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.835581 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835566 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 22:23:10.835645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835578 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.835645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835582 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.835645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835626 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.835645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835610 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.835645 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835631 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.835843 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.835632 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 22:23:10.837982 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.837966 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5j2lt"] Nov 01 22:23:10.838073 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.837993 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jpxjw"] Nov 01 22:23:10.838073 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.838007 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtzbw"] Nov 01 22:23:10.838073 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.838021 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b8b6447f-8bbxr"] Nov 01 22:23:10.838211 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.838096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:10.839232 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.839209 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 22:23:10.839623 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.839606 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 22:23:10.842801 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.842780 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 22:23:10.842913 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.842881 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.842969 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.842914 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.851980 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.851963 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8zwrr"] Nov 01 22:23:10.852099 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.852084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.871686 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.871665 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 22:23:10.871686 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.871679 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 22:23:10.871842 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.871676 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 22:23:10.871842 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.871681 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 22:23:10.871842 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.871669 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 22:23:10.875423 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.875409 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wn5v5"] Nov 01 22:23:10.875544 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.875529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:10.879051 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.879037 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:10.879119 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.879063 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:10.884781 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.884879 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpx7\" (UniqueName: \"kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7\") pod \"kas-connection-checker-685895c7c9-jpxjw\" (UID: \"06fbbe23-1f70-4c1e-b82f-5fa72a4166db\") " pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:10.884879 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.884879 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.885020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9mz\" (UniqueName: \"kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.885020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.884993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.885162 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmds\" (UniqueName: \"kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.885283 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.885300 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5b5746c-v88s9: secret "image-registry-tls" not found Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.885371 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls podName:24621246-44ab-4de5-a8a8-389f98b09d22 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.385353575 +0000 UTC m=+34.182129034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls") pod "image-registry-5bb5b5746c-v88s9" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22") : secret "image-registry-tls" not found Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.885548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrqx\" (UniqueName: \"kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885730 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p689n\" (UniqueName: \"kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n\") pod \"kas-connection-checker-685895c7c9-5j2lt\" (UID: \"65225fe4-74e2-4b06-9169-7dce163b2fec\") " pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmz4\" (UniqueName: \"kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.885974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.885958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69zf\" (UniqueName: \"kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf\") pod \"kas-connection-checker-685895c7c9-wtzbw\" (UID: \"29c2c4f7-79ec-444c-be78-ce43edf6850c\") " pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92dfm\" (UniqueName: \"kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886164 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.886448 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:10.886536 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.886508 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.386493659 +0000 UTC m=+34.183269116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:10.887029 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.886562 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:10.887029 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.886595 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.386585633 +0000 UTC m=+34.183361089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:10.887029 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886784 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:10.887029 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886872 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.887029 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.886890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.889501 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.889484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.889554 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.889521 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.889746 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.889730 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb"] Nov 01 22:23:10.889845 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.889831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:10.894121 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.894095 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpx7\" (UniqueName: \"kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7\") pod \"kas-connection-checker-685895c7c9-jpxjw\" (UID: \"06fbbe23-1f70-4c1e-b82f-5fa72a4166db\") " pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:10.895128 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.895108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.895406 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.895387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrqx\" (UniqueName: \"kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:10.895454 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.895387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:10.896951 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.896931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69zf\" (UniqueName: \"kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf\") pod \"kas-connection-checker-685895c7c9-wtzbw\" (UID: \"29c2c4f7-79ec-444c-be78-ce43edf6850c\") " pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:10.896951 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.896946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p689n\" (UniqueName: \"kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n\") pod \"kas-connection-checker-685895c7c9-5j2lt\" (UID: \"65225fe4-74e2-4b06-9169-7dce163b2fec\") " pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:10.925729 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.925704 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc"] Nov 01 22:23:10.925834 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.925819 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:10.936043 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936014 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.936043 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936023 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.936043 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936028 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:10.936240 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936080 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.936240 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936176 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 22:23:10.936240 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936179 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.936240 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936197 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:10.936240 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.936235 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:10.951546 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951519 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz"] Nov 01 22:23:10.951546 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951547 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls"] Nov 01 22:23:10.951662 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951561 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nkrwh"] Nov 01 22:23:10.951662 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951574 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nbf7s"] Nov 01 22:23:10.951662 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951588 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7"] Nov 01 22:23:10.951662 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951600 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx"] Nov 01 22:23:10.951662 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951619 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:23:10.951957 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.951937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:10.952020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952004 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2rlrx"] Nov 01 22:23:10.952085 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952073 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8zwrr"] Nov 01 22:23:10.952132 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952121 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc"] Nov 01 22:23:10.952175 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952143 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:23:10.952175 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952173 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wn5v5"] Nov 01 22:23:10.952262 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952206 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b8b6447f-8bbxr"] Nov 01 22:23:10.952262 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.952245 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb"] Nov 01 22:23:10.959715 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.959690 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:10.971506 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.971485 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:10.971617 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.971604 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:10.982328 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.982308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:10.986941 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.986921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.987046 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.986950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:10.987046 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.986970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.987046 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.986986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.987176 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.987176 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9mz\" (UniqueName: \"kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.987176 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.987181 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2hg\" (UniqueName: \"kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.987195 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: secret "image-registry-tls" not found Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:10.987249 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.487227018 +0000 UTC m=+34.284002477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : secret "image-registry-tls" not found Nov 01 22:23:10.987312 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46dx\" (UniqueName: \"kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmds\" (UniqueName: \"kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.987595 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.987567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.988368 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.988453 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.988453 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.988560 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.988560 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.988667 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.988745 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.988796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.988957 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:10.988998 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmz4\" (UniqueName: \"kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:10.988998 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.988988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:10.989050 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.989050 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:10.989110 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7jk\" (UniqueName: \"kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:10.989232 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwbk\" (UniqueName: \"kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:10.989265 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:10.989293 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92dfm\" (UniqueName: \"kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:10.989327 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:10.989327 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.989318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:10.999511 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.999214 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:10.999511 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.999310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:10.999511 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:10.999322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.000093 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:11.000170 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:11.000361 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000342 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.000629 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:11.000700 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000681 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:11.000841 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.000817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.001488 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.001442 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.001566 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.001519 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.002533 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.002504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.004264 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.003998 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:11.004367 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.004344 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.004921 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.004833 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:11.005349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.005324 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.005431 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.005404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.005482 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.005429 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9mz\" (UniqueName: \"kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:11.005780 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.005760 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmz4\" (UniqueName: \"kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:11.005908 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.005885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.006099 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.006078 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmds\" (UniqueName: \"kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.006796 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.006776 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92dfm\" (UniqueName: \"kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:11.006865 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.006815 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:11.007068 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.007047 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.036757 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.036735 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:11.036912 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.036898 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:11.059409 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.059237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46dx\" (UniqueName: \"kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.089906 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.089938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.089969 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.589951248 +0000 UTC m=+34.386726709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7jk\" (UniqueName: \"kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwbk\" (UniqueName: \"kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltb82\" (UniqueName: \"kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82\") pod \"volume-data-source-validator-58c9f5f46d-znttb\" (UID: \"3ad608dd-a3a4-47cb-9702-4cbef303f2f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvh6m\" (UniqueName: \"kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m\") pod \"network-check-source-6f78d59f99-xqcfc\" (UID: \"e98acdf9-450d-46a8-b3e8-faae9509a855\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:11.092354 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.090292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2hg\" (UniqueName: \"kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.091011 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.590992099 +0000 UTC m=+34.387767556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.091186 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.091245 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.591230818 +0000 UTC m=+34.388006274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.091664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.091764 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:11.093087 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.091821 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.591805153 +0000 UTC m=+34.388580594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:11.094044 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.093601 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.094044 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.093985 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.095030 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.095001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.095624 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.095580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.096204 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.096185 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:11.102887 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.102807 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46dx\" (UniqueName: \"kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.103902 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.103794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwbk\" (UniqueName: \"kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:11.104368 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.104346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:11.106238 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.106208 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2hg\" (UniqueName: \"kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.109714 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.109692 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7jk\" (UniqueName: \"kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.124225 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.121326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:11.132120 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.132092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:11.191142 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.191112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltb82\" (UniqueName: \"kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82\") pod \"volume-data-source-validator-58c9f5f46d-znttb\" (UID: \"3ad608dd-a3a4-47cb-9702-4cbef303f2f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:11.191276 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.191153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvh6m\" (UniqueName: \"kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m\") pod \"network-check-source-6f78d59f99-xqcfc\" (UID: \"e98acdf9-450d-46a8-b3e8-faae9509a855\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:11.201146 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.201008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltb82\" (UniqueName: \"kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82\") pod \"volume-data-source-validator-58c9f5f46d-znttb\" (UID: \"3ad608dd-a3a4-47cb-9702-4cbef303f2f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:11.201215 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.201121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvh6m\" (UniqueName: \"kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m\") pod \"network-check-source-6f78d59f99-xqcfc\" (UID: \"e98acdf9-450d-46a8-b3e8-faae9509a855\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:11.287369 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.287335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:11.309937 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.309918 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jpxjw"] Nov 01 22:23:11.313401 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.313373 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fbbe23_1f70_4c1e_b82f_5fa72a4166db.slice/crio-d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2 WatchSource:0}: Error finding container d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2: Status 404 returned error can't find the container with id d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2 Nov 01 22:23:11.319557 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.319532 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5j2lt"] Nov 01 22:23:11.320352 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.320328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:11.320897 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.320871 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtzbw"] Nov 01 22:23:11.321033 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.321013 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c2c4f7_79ec_444c_be78_ce43edf6850c.slice/crio-88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d WatchSource:0}: Error finding container 88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d: Status 404 returned error can't find the container with id 88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d Nov 01 22:23:11.347524 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.347498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:11.380388 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.380223 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7"] Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.394494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.394663 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.394728 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.394709473 +0000 UTC m=+35.191484938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395393 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395435 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395445 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395465 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5b5746c-v88s9: secret "image-registry-tls" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395469 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret podName:9445ebf3-feba-48ea-8eb6-882fd83343ab nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.395449387 +0000 UTC m=+66.192224834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret") pod "global-pull-secret-syncer-vhgqk" (UID: "9445ebf3-feba-48ea-8eb6-882fd83343ab") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395497 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.395481226 +0000 UTC m=+35.192256668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:11.395538 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395517 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls podName:24621246-44ab-4de5-a8a8-389f98b09d22 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.39550543 +0000 UTC m=+35.192280912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls") pod "image-registry-5bb5b5746c-v88s9" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22") : secret "image-registry-tls" not found Nov 01 22:23:11.396348 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395547 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:11.396348 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.395816 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs podName:57c7b714-b4a8-40b1-a4ec-2c0a2117e21c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.395802046 +0000 UTC m=+66.192577491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs") pod "network-metrics-daemon-jhdh2" (UID: "57c7b714-b4a8-40b1-a4ec-2c0a2117e21c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:11.403048 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.403015 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e707915_61a5_42f8_bb48_920bda324148.slice/crio-bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93 WatchSource:0}: Error finding container bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93: Status 404 returned error can't find the container with id bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93 Nov 01 22:23:11.403741 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.403715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:11.405293 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.405203 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nbf7s"] Nov 01 22:23:11.419840 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.419807 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5e010_5d31_46f0_bb5a_a952613af986.slice/crio-6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b WatchSource:0}: Error finding container 6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b: Status 404 returned error can't find the container with id 6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b Nov 01 22:23:11.428323 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.428260 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nkrwh"] Nov 01 22:23:11.440236 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.440194 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47d3a1e_edf7_4b20_b30c_96449721dbef.slice/crio-bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e WatchSource:0}: Error finding container bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e: Status 404 returned error can't find the container with id bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e Nov 01 22:23:11.447117 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.447073 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls"] Nov 01 22:23:11.457020 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.456985 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c627c44_b90e_4477_ade0_1a9ce7aafb36.slice/crio-d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610 WatchSource:0}: Error finding container d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610: Status 404 returned error can't find the container with id d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610 Nov 01 22:23:11.498375 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.497806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:11.498375 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.498015 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:11.498375 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.498033 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: secret "image-registry-tls" not found Nov 01 22:23:11.498375 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.498090 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.498072553 +0000 UTC m=+35.294848015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : secret "image-registry-tls" not found Nov 01 22:23:11.569264 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.569241 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wn5v5"] Nov 01 22:23:11.572272 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.572249 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f57d53_5511_4fc4_86b1_a4884a4d213a.slice/crio-c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14 WatchSource:0}: Error finding container c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14: Status 404 returned error can't find the container with id c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14 Nov 01 22:23:11.598763 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.598604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:11.598843 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598758 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:11.598843 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.598776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:11.598843 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598811 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.598793949 +0000 UTC m=+35.395569394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598868 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.598894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598904 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.598894311 +0000 UTC m=+35.395669753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.598945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598963 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.598951945 +0000 UTC m=+35.395727401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:11.599000 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.598982 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:11.599213 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:11.599018 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:12.599011559 +0000 UTC m=+35.395787000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:11.656489 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.656462 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc"] Nov 01 22:23:11.659123 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.659101 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98acdf9_450d_46a8_b3e8_faae9509a855.slice/crio-07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8 WatchSource:0}: Error finding container 07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8: Status 404 returned error can't find the container with id 07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8 Nov 01 22:23:11.692760 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.692734 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb"] Nov 01 22:23:11.695439 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:11.695414 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad608dd_a3a4_47cb_9702_4cbef303f2f7.slice/crio-8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7 WatchSource:0}: Error finding container 8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7: Status 404 returned error can't find the container with id 8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7 Nov 01 22:23:11.720867 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.720834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:11.721088 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.721069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:11.721248 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.721227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:11.725216 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.725197 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:11.725216 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.725189 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:11.725907 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.725235 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:11.725907 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.725246 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:11.725907 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.725253 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:11.797487 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.797466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:11.993647 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.993601 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wn5v5" event={"ID":"49f57d53-5511-4fc4-86b1-a4884a4d213a","Type":"ContainerStarted","Data":"c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14"} Nov 01 22:23:11.994917 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.994887 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" event={"ID":"3ad608dd-a3a4-47cb-9702-4cbef303f2f7","Type":"ContainerStarted","Data":"8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7"} Nov 01 22:23:11.996075 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.996051 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" event={"ID":"e98acdf9-450d-46a8-b3e8-faae9509a855","Type":"ContainerStarted","Data":"07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8"} Nov 01 22:23:11.997109 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.997076 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" event={"ID":"6c627c44-b90e-4477-ade0-1a9ce7aafb36","Type":"ContainerStarted","Data":"d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610"} Nov 01 22:23:11.998236 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.998210 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" event={"ID":"d47d3a1e-edf7-4b20-b30c-96449721dbef","Type":"ContainerStarted","Data":"bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e"} Nov 01 22:23:11.999683 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.999666 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" event={"ID":"29c2c4f7-79ec-444c-be78-ce43edf6850c","Type":"ContainerStarted","Data":"5af821bdfc34a8a1de420d99ec46ccbf87a4491d9a123b90cb07c544f74cf914"} Nov 01 22:23:11.999756 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:11.999693 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" event={"ID":"29c2c4f7-79ec-444c-be78-ce43edf6850c","Type":"ContainerStarted","Data":"88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d"} Nov 01 22:23:12.001038 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.001013 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" event={"ID":"8aa5e010-5d31-46f0-bb5a-a952613af986","Type":"ContainerStarted","Data":"6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b"} Nov 01 22:23:12.002087 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.002064 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" event={"ID":"0e707915-61a5-42f8-bb48-920bda324148","Type":"ContainerStarted","Data":"bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93"} Nov 01 22:23:12.003583 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.003563 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" event={"ID":"06fbbe23-1f70-4c1e-b82f-5fa72a4166db","Type":"ContainerStarted","Data":"53fed8158071d66cfbd6ba3c6150579e561b9d453ad2a095825cdc2cfc6ffe36"} Nov 01 22:23:12.003686 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.003603 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" event={"ID":"06fbbe23-1f70-4c1e-b82f-5fa72a4166db","Type":"ContainerStarted","Data":"d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2"} Nov 01 22:23:12.005116 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.005093 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" event={"ID":"65225fe4-74e2-4b06-9169-7dce163b2fec","Type":"ContainerStarted","Data":"37f96cd8cc2606c5ca1984c413f3db4e019d867ee3576738c5a518e9c31f1a66"} Nov 01 22:23:12.005206 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.005122 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" event={"ID":"65225fe4-74e2-4b06-9169-7dce163b2fec","Type":"ContainerStarted","Data":"5064614125a8f7d8f78041e5087a6f44cd66abaf4166821462e8366885397acb"} Nov 01 22:23:12.017022 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.016985 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" podStartSLOduration=449.016973244 podStartE2EDuration="7m29.016973244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:12.015893434 +0000 UTC m=+34.812668900" watchObservedRunningTime="2026-11-01 22:23:12.016973244 +0000 UTC m=+34.813748707" Nov 01 22:23:12.032574 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.031611 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" podStartSLOduration=449.031595106 podStartE2EDuration="7m29.031595106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:12.030899814 +0000 UTC m=+34.827675275" watchObservedRunningTime="2026-11-01 22:23:12.031595106 +0000 UTC m=+34.828370571" Nov 01 22:23:12.080226 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.080169 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" podStartSLOduration=449.080154641 podStartE2EDuration="7m29.080154641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:12.050322275 +0000 UTC m=+34.847097739" watchObservedRunningTime="2026-11-01 22:23:12.080154641 +0000 UTC m=+34.876930131" Nov 01 22:23:12.080372 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.080302 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvpsn"] Nov 01 22:23:12.085140 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:12.085108 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ca1661_6040_419a_9037_1241908c41fa.slice/crio-400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5 WatchSource:0}: Error finding container 400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5: Status 404 returned error can't find the container with id 400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5 Nov 01 22:23:12.408101 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.408053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:12.408283 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.408129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:12.408283 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.408218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:12.408283 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408225 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:12.408283 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408249 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5b5746c-v88s9: secret "image-registry-tls" not found Nov 01 22:23:12.408485 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408315 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls podName:24621246-44ab-4de5-a8a8-389f98b09d22 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.408294205 +0000 UTC m=+37.205069668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls") pod "image-registry-5bb5b5746c-v88s9" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22") : secret "image-registry-tls" not found Nov 01 22:23:12.408485 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408371 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:12.408485 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.408411307 +0000 UTC m=+37.205186766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:12.408699 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408490 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:12.408699 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.408522 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.40851096 +0000 UTC m=+37.205286401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:12.509200 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.509162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:12.509370 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.509341 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:12.509370 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.509365 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: secret "image-registry-tls" not found Nov 01 22:23:12.509444 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.509437 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.509416705 +0000 UTC m=+37.306192154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : secret "image-registry-tls" not found Nov 01 22:23:12.610958 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.610914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:12.611129 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.611054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:12.611129 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611063 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:12.611129 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.611122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:12.611301 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611135 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.61111553 +0000 UTC m=+37.407890985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:12.611301 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611210 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:12.611301 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:12.611215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:12.611301 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611261 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.611246291 +0000 UTC m=+37.408021747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:12.611486 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611324 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:12.611486 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611336 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.611323436 +0000 UTC m=+37.408098892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:12.611486 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:12.611358 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.611346776 +0000 UTC m=+37.408122222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:13.015697 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:13.015660 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvpsn" event={"ID":"d5ca1661-6040-419a-9037-1241908c41fa","Type":"ContainerStarted","Data":"400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5"} Nov 01 22:23:14.023298 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.023212 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" event={"ID":"3ad608dd-a3a4-47cb-9702-4cbef303f2f7","Type":"ContainerStarted","Data":"a5839c94a09f5defeabd84796149f008929571be5db7b6ba7254a333e417d283"} Nov 01 22:23:14.039552 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.039497 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" podStartSLOduration=441.970338246 podStartE2EDuration="7m24.039478305s" totalImagesPullingTime="2.069140059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:14.037995268 +0000 UTC m=+36.834770726" watchObservedRunningTime="2026-11-01 22:23:14.039478305 +0000 UTC m=+36.836253768" Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.429090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.429217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.429247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429362 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429411 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.429398723 +0000 UTC m=+41.226174166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429757 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429795 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.42978419 +0000 UTC m=+41.226559632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429840 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429867 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5b5746c-v88s9: secret "image-registry-tls" not found Nov 01 22:23:14.429922 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.429892 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls podName:24621246-44ab-4de5-a8a8-389f98b09d22 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.429883581 +0000 UTC m=+41.226659022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls") pod "image-registry-5bb5b5746c-v88s9" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22") : secret "image-registry-tls" not found Nov 01 22:23:14.531363 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.530699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:14.531363 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.530906 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:14.531363 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.530923 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: secret "image-registry-tls" not found Nov 01 22:23:14.531363 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.530984 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.53096359 +0000 UTC m=+41.327739037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : secret "image-registry-tls" not found Nov 01 22:23:14.632306 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.632265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:14.632504 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.632361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:14.632504 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.632415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:14.632504 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:14.632498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:14.632661 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.632619 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:14.632710 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.632677 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.632660799 +0000 UTC m=+41.429436247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:14.632966 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.632946 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.632930254 +0000 UTC m=+41.429705696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:14.633086 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.633022 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:14.633086 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.633055 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.633044688 +0000 UTC m=+41.429820144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:14.633185 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.633104 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:14.633185 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:14.633135 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.633124908 +0000 UTC m=+41.429900364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:16.035925 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.035699 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" event={"ID":"d47d3a1e-edf7-4b20-b30c-96449721dbef","Type":"ContainerStarted","Data":"59a0189d8e164b160e0c396d27afc7ae0f98afdfa4a3189f06453c489fc65199"} Nov 01 22:23:16.039271 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.039202 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/0.log" Nov 01 22:23:16.039818 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.039791 2573 scope.go:122] "RemoveContainer" containerID="880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad" Nov 01 22:23:16.040018 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.039780 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" event={"ID":"8aa5e010-5d31-46f0-bb5a-a952613af986","Type":"ContainerStarted","Data":"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad"} Nov 01 22:23:16.056519 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.056199 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" podStartSLOduration=442.68616248 podStartE2EDuration="7m27.056180179s" totalImagesPullingTime="4.370017699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:16.054072202 +0000 UTC m=+38.850847681" watchObservedRunningTime="2026-11-01 22:23:16.056180179 +0000 UTC m=+38.852955645" Nov 01 22:23:16.650539 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.650500 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2jgr7"] Nov 01 22:23:16.694883 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.694118 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jgr7"] Nov 01 22:23:16.694883 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.694231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.697786 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.697763 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:16.698974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.698928 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:16.698974 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.698933 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:16.858048 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.858003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.858215 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.858085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.858282 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.858199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.858334 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.858314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.858555 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.858377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksncs\" (UniqueName: \"kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.959792 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.959756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960014 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.959842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksncs\" (UniqueName: \"kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960014 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:16.959959 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:16.960125 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.959958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960125 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:16.960037 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:17.460017077 +0000 UTC m=+40.256792519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:16.960125 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.960102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960274 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.960139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960274 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.960189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.960274 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:16.960199 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:16.960274 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:16.960257 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:17.46023871 +0000 UTC m=+40.257014193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : configmap "kube-rbac-proxy" not found Nov 01 22:23:16.960528 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.960493 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:16.973581 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:16.973523 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksncs\" (UniqueName: \"kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:17.045085 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.045052 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:17.046106 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.045569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/0.log" Nov 01 22:23:17.046106 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.045599 2573 generic.go:356] "Generic (PLEG): container finished" podID="8aa5e010-5d31-46f0-bb5a-a952613af986" containerID="880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad" exitCode=255 Nov 01 22:23:17.046106 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.045613 2573 generic.go:356] "Generic (PLEG): container finished" podID="8aa5e010-5d31-46f0-bb5a-a952613af986" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" exitCode=255 Nov 01 22:23:17.046261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.046166 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" event={"ID":"8aa5e010-5d31-46f0-bb5a-a952613af986","Type":"ContainerDied","Data":"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad"} Nov 01 22:23:17.046261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.046212 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" event={"ID":"8aa5e010-5d31-46f0-bb5a-a952613af986","Type":"ContainerDied","Data":"ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7"} Nov 01 22:23:17.046261 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.046239 2573 scope.go:122] "RemoveContainer" containerID="880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad" Nov 01 22:23:17.046660 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.046643 2573 scope.go:122] "RemoveContainer" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" Nov 01 22:23:17.047079 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.046872 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nbf7s_openshift-console-operator(8aa5e010-5d31-46f0-bb5a-a952613af986)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" podUID="8aa5e010-5d31-46f0-bb5a-a952613af986" Nov 01 22:23:17.219965 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.219722 2573 scope.go:122] "RemoveContainer" containerID="880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad" Nov 01 22:23:17.220450 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.220357 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad\": container with ID starting with 880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad not found: ID does not exist" containerID="880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad" Nov 01 22:23:17.220450 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.220406 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad"} err="failed to get container status \"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad\": rpc error: code = NotFound desc = could not find container \"880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad\": container with ID starting with 880053d6cc48bcd586cc774f5cf2327116012c07b34e0ad44ce8c25e13dffcad not found: ID does not exist" Nov 01 22:23:17.466173 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.466129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:17.466350 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.466299 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:17.466399 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.466358 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.466345111 +0000 UTC m=+41.263120552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : configmap "kube-rbac-proxy" not found Nov 01 22:23:17.466458 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.466432 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:17.466589 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:17.466509 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.466491716 +0000 UTC m=+41.263267158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:17.466690 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:17.466208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:18.051634 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.051305 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" event={"ID":"6c627c44-b90e-4477-ade0-1a9ce7aafb36","Type":"ContainerStarted","Data":"7c56edabb3eefb4b2292e135c9bdec28eaedcdcfae0451b5cfecee3bd3741f2a"} Nov 01 22:23:18.053503 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.053478 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:18.053825 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.053794 2573 scope.go:122] "RemoveContainer" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" Nov 01 22:23:18.054049 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.054013 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nbf7s_openshift-console-operator(8aa5e010-5d31-46f0-bb5a-a952613af986)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" podUID="8aa5e010-5d31-46f0-bb5a-a952613af986" Nov 01 22:23:18.055180 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.055115 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wn5v5" event={"ID":"49f57d53-5511-4fc4-86b1-a4884a4d213a","Type":"ContainerStarted","Data":"075b5527a2a49cbef20d135283418b4b8dc5512331eaaaa91b6f29203d487043"} Nov 01 22:23:18.068776 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.068715 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" podStartSLOduration=420.80026402 podStartE2EDuration="7m7.068694854s" totalImagesPullingTime="6.268430834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:18.066537887 +0000 UTC m=+40.863313353" watchObservedRunningTime="2026-11-01 22:23:18.068694854 +0000 UTC m=+40.865470320" Nov 01 22:23:18.082723 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.082666 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wn5v5" podStartSLOduration=2.331885098 podStartE2EDuration="8.082652216s" totalImagesPullingTime="5.750767118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:18.08149866 +0000 UTC m=+40.878274127" watchObservedRunningTime="2026-11-01 22:23:18.082652216 +0000 UTC m=+40.879427681" Nov 01 22:23:18.478424 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.478384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.478468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.478517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.478545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478552 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.478581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:18.478608 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478590 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478620 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478631 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.47861085 +0000 UTC m=+49.275386310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478634 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5b5746c-v88s9: secret "image-registry-tls" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478670 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478672 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:20.478659122 +0000 UTC m=+43.275434567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : configmap "kube-rbac-proxy" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478691 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls podName:24621246-44ab-4de5-a8a8-389f98b09d22 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.478683872 +0000 UTC m=+49.275459318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls") pod "image-registry-5bb5b5746c-v88s9" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22") : secret "image-registry-tls" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478707 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.478699719 +0000 UTC m=+49.275475167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478715 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:18.478924 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.478770 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:20.478752878 +0000 UTC m=+43.275528338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:18.579699 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.579662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:18.579903 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.579815 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 22:23:18.579903 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.579828 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: secret "image-registry-tls" not found Nov 01 22:23:18.579903 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.579895 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.579879208 +0000 UTC m=+49.376654653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : secret "image-registry-tls" not found Nov 01 22:23:18.680886 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.680830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:18.681054 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.680954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:18.681054 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.680978 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:18.681054 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.681029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:18.681054 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681041 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.681024959 +0000 UTC m=+49.477800416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681099 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681105 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:18.681119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681153 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.681136995 +0000 UTC m=+49.477912458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681185 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.681175477 +0000 UTC m=+49.477950923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:18.681279 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:18.681203 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:26.681193962 +0000 UTC m=+49.477969408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:19.060875 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.060766 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" event={"ID":"e98acdf9-450d-46a8-b3e8-faae9509a855","Type":"ContainerStarted","Data":"1987fe7dee0c9e27d9a2716b8438eda5dfaf93aeb2da0d98b24d5d1d2498cb4f"} Nov 01 22:23:19.062468 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.062438 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" event={"ID":"0e707915-61a5-42f8-bb48-920bda324148","Type":"ContainerStarted","Data":"854f193bce62849627de13556755f7c3f3ce3962dea2077cc720152cbc94ce8d"} Nov 01 22:23:19.064345 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.064318 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvpsn" event={"ID":"d5ca1661-6040-419a-9037-1241908c41fa","Type":"ContainerStarted","Data":"e2fb838d0ce05d84ce47dbd127c0c4537ad7ad19d7e1cda3dd9e9181dd2b8dd0"} Nov 01 22:23:19.064522 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.064388 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:19.094377 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.094293 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" podStartSLOduration=417.543424659 podStartE2EDuration="7m4.094275747s" totalImagesPullingTime="6.550851088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:19.092285763 +0000 UTC m=+41.889061226" watchObservedRunningTime="2026-11-01 22:23:19.094275747 +0000 UTC m=+41.891051216" Nov 01 22:23:19.120058 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.120008 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" podStartSLOduration=442.795293916 podStartE2EDuration="7m30.119994433s" totalImagesPullingTime="7.324700517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:19.118760152 +0000 UTC m=+41.915535620" watchObservedRunningTime="2026-11-01 22:23:19.119994433 +0000 UTC m=+41.916769891" Nov 01 22:23:19.142164 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.142115 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zvpsn" podStartSLOduration=36.031392631 podStartE2EDuration="42.142101914s" totalImagesPullingTime="6.110709283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:19.141011066 +0000 UTC m=+41.937786530" watchObservedRunningTime="2026-11-01 22:23:19.142101914 +0000 UTC m=+41.938877377" Nov 01 22:23:19.445185 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.445151 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg"] Nov 01 22:23:19.467026 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.467001 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg"] Nov 01 22:23:19.467183 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.467100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:19.471097 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.471071 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:19.471241 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.471106 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 22:23:19.565603 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.565570 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d6jqb\"" Nov 01 22:23:19.590345 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.590319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjk4g\" (UniqueName: \"kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g\") pod \"migrator-66df5d496b-h9fsg\" (UID: \"a6acb24b-4146-452f-b222-008b883c77d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:19.691699 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.691668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk4g\" (UniqueName: \"kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g\") pod \"migrator-66df5d496b-h9fsg\" (UID: \"a6acb24b-4146-452f-b222-008b883c77d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:19.703151 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.703098 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk4g\" (UniqueName: \"kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g\") pod \"migrator-66df5d496b-h9fsg\" (UID: \"a6acb24b-4146-452f-b222-008b883c77d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:19.791704 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:19.791675 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:20.080253 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:20.080019 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg"] Nov 01 22:23:20.083721 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:20.083694 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6acb24b_4146_452f_b222_008b883c77d0.slice/crio-8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6 WatchSource:0}: Error finding container 8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6: Status 404 returned error can't find the container with id 8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6 Nov 01 22:23:20.499599 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:20.499569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:20.499782 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:20.499639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:20.499782 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:20.499740 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:20.499873 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:20.499822 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:24.499800222 +0000 UTC m=+47.296575664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:20.500287 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:20.500271 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:21.072826 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:21.072784 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" event={"ID":"a6acb24b-4146-452f-b222-008b883c77d0","Type":"ContainerStarted","Data":"8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6"} Nov 01 22:23:21.105304 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:21.105270 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:21.106001 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:21.105324 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:21.106001 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:21.105773 2573 scope.go:122] "RemoveContainer" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" Nov 01 22:23:21.106104 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:21.106013 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nbf7s_openshift-console-operator(8aa5e010-5d31-46f0-bb5a-a952613af986)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" podUID="8aa5e010-5d31-46f0-bb5a-a952613af986" Nov 01 22:23:22.077755 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:22.077556 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" event={"ID":"a6acb24b-4146-452f-b222-008b883c77d0","Type":"ContainerStarted","Data":"13bf215f1c6836b4c230a2f0e3e6d262640fd20c44a3265ba6d6d6e671c86f85"} Nov 01 22:23:22.092964 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:22.092916 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" podStartSLOduration=1.195039895 podStartE2EDuration="3.092902405s" totalImagesPullingTime="1.89786251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:22.09238965 +0000 UTC m=+44.889165113" watchObservedRunningTime="2026-11-01 22:23:22.092902405 +0000 UTC m=+44.889677869" Nov 01 22:23:22.871482 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:22.871453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5wh4_d2372e60-3b4e-407f-b0ed-6799496e9dd7/dns-node-resolver/0.log" Nov 01 22:23:24.071621 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:24.071598 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xpgg7_41572fd2-3250-4c36-a7be-8c8f733c39c3/node-ca/0.log" Nov 01 22:23:24.536045 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:24.536000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:24.536199 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:24.536154 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:24.536257 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:24.536221 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:32.536206452 +0000 UTC m=+55.332981894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:24.671700 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:24.671677 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wn5v5_49f57d53-5511-4fc4-86b1-a4884a4d213a/serve-healthcheck-canary/0.log" Nov 01 22:23:25.072254 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:25.072223 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-h9fsg_a6acb24b-4146-452f-b222-008b883c77d0/migrator/0.log" Nov 01 22:23:25.273053 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:25.273025 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7_0e707915-61a5-42f8-bb48-920bda324148/kube-storage-version-migrator-operator/0.log" Nov 01 22:23:26.554226 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.554173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.554288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.554335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.554346 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.554409 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls podName:bc99826a-2d2c-41ae-b55a-e95475056ead nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.554395544 +0000 UTC m=+65.351170986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qsgwz" (UID: "bc99826a-2d2c-41ae-b55a-e95475056ead") : secret "cluster-monitoring-operator-tls" not found Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.554433 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 22:23:26.554635 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.554480 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert podName:1681c152-d4a7-417f-a54e-a7a4b11a2575 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.554469333 +0000 UTC m=+65.351244775 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2rlrx" (UID: "1681c152-d4a7-417f-a54e-a7a4b11a2575") : secret "networking-console-plugin-cert" not found Nov 01 22:23:26.557910 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.557891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:26.617753 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.617726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:26.655497 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.655453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:26.658132 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.658108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:26.746682 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.746645 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:26.755946 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.755897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:26.756020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.755952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:26.756020 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.756000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:26.756123 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756056 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 22:23:26.756123 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.756055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:26.756213 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756118 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 22:23:26.756213 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756137 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls podName:a8039013-0ab2-45d6-9321-da3ceb7dc478 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.75611105 +0000 UTC m=+65.552886512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-msljx" (UID: "a8039013-0ab2-45d6-9321-da3ceb7dc478") : secret "samples-operator-tls" not found Nov 01 22:23:26.756213 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756155 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.756146856 +0000 UTC m=+65.552922301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : configmap references non-existent config key: service-ca.crt Nov 01 22:23:26.756213 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756169 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs podName:a0650f02-2be4-4fad-bd28-2fa18f86c6af nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.756161077 +0000 UTC m=+65.552936523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs") pod "router-default-7b8b6447f-8bbxr" (UID: "a0650f02-2be4-4fad-bd28-2fa18f86c6af") : secret "router-metrics-certs-default" not found Nov 01 22:23:26.756425 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756307 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:26.756482 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:26.756426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls podName:010a5ac6-aeef-45ce-8c57-dcebf29427b1 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.756410939 +0000 UTC m=+65.553186394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls") pod "dns-default-8zwrr" (UID: "010a5ac6-aeef-45ce-8c57-dcebf29427b1") : secret "dns-default-metrics-tls" not found Nov 01 22:23:26.894349 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:26.894302 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:23:26.897158 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:26.897019 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24621246_44ab_4de5_a8a8_389f98b09d22.slice/crio-038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed WatchSource:0}: Error finding container 038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed: Status 404 returned error can't find the container with id 038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed Nov 01 22:23:27.020713 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.020683 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:23:27.040586 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:27.040559 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c3582e_4821_483f_b90a_6175bd9e27be.slice/crio-39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd WatchSource:0}: Error finding container 39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd: Status 404 returned error can't find the container with id 39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd Nov 01 22:23:27.095744 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.095712 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" event={"ID":"24621246-44ab-4de5-a8a8-389f98b09d22","Type":"ContainerStarted","Data":"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897"} Nov 01 22:23:27.095885 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.095777 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" event={"ID":"24621246-44ab-4de5-a8a8-389f98b09d22","Type":"ContainerStarted","Data":"038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed"} Nov 01 22:23:27.095954 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.095919 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:27.097612 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.097578 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" event={"ID":"d8c3582e-4821-483f-b90a-6175bd9e27be","Type":"ContainerStarted","Data":"39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd"} Nov 01 22:23:27.120309 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.120259 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" podStartSLOduration=46.120245171 podStartE2EDuration="46.120245171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:27.119350743 +0000 UTC m=+49.916126208" watchObservedRunningTime="2026-11-01 22:23:27.120245171 +0000 UTC m=+49.917020635" Nov 01 22:23:27.722334 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:27.722305 2573 scope.go:122] "RemoveContainer" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" Nov 01 22:23:28.105317 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.105223 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" event={"ID":"d8c3582e-4821-483f-b90a-6175bd9e27be","Type":"ContainerStarted","Data":"449549da8aa4e9704cf2b2618796939b2cf090c3e1221648c916ab2849199701"} Nov 01 22:23:28.105474 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.105351 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:28.107135 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.107115 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:28.107263 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.107219 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" event={"ID":"8aa5e010-5d31-46f0-bb5a-a952613af986","Type":"ContainerStarted","Data":"522ab13ea2ba6d5aae04bd4fc5311f6235bc6cf4bf484b0893329381e20271b4"} Nov 01 22:23:28.107522 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.107490 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:28.112061 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.112045 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:28.125617 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.125575 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" podStartSLOduration=437.125560746 podStartE2EDuration="7m17.125560746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:28.125067678 +0000 UTC m=+50.921843168" watchObservedRunningTime="2026-11-01 22:23:28.125560746 +0000 UTC m=+50.922336214" Nov 01 22:23:28.141533 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:28.141500 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" podStartSLOduration=432.767680824 podStartE2EDuration="7m17.141489804s" totalImagesPullingTime="4.37380898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:28.140762172 +0000 UTC m=+50.937537659" watchObservedRunningTime="2026-11-01 22:23:28.141489804 +0000 UTC m=+50.938265268" Nov 01 22:23:32.608018 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:32.607972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:32.608570 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:32.608136 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:32.608570 ip-10-0-141-223 kubenswrapper[2573]: E1101 22:23:32.608199 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls podName:63988932-7675-4400-beb1-610e2ecfde16 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:48.608182957 +0000 UTC m=+71.404958413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jgr7" (UID: "63988932-7675-4400-beb1-610e2ecfde16") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:34.984250 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:34.984222 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:40.312332 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:40.312295 2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:23:42.593548 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.593488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:42.594013 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.593596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:42.596106 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.596069 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:42.596245 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.596222 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:42.794921 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.794884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:42.795145 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.794953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:42.795145 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.795008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:42.795239 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.795154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:42.795793 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.795705 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:42.797832 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.797796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:42.797956 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.797874 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:42.797956 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.797892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:42.837801 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.837772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:42.884571 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.884349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:42.982049 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:42.982011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:43.020181 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.020141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:43.064958 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.064931 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:43.073044 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.073009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:43.112268 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.112051 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz"] Nov 01 22:23:43.114905 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.114877 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc99826a_2d2c_41ae_b55a_e95475056ead.slice/crio-22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0 WatchSource:0}: Error finding container 22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0: Status 404 returned error can't find the container with id 22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0 Nov 01 22:23:43.154290 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.154261 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" event={"ID":"bc99826a-2d2c-41ae-b55a-e95475056ead","Type":"ContainerStarted","Data":"22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0"} Nov 01 22:23:43.158676 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.158655 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2rlrx"] Nov 01 22:23:43.160891 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.160837 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1681c152_d4a7_417f_a54e_a7a4b11a2575.slice/crio-97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b WatchSource:0}: Error finding container 97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b: Status 404 returned error can't find the container with id 97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b Nov 01 22:23:43.255341 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.255319 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx"] Nov 01 22:23:43.298828 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.298803 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b8b6447f-8bbxr"] Nov 01 22:23:43.301078 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.301055 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0650f02_2be4_4fad_bd28_2fa18f86c6af.slice/crio-4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b WatchSource:0}: Error finding container 4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b: Status 404 returned error can't find the container with id 4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b Nov 01 22:23:43.352465 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.352432 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8zwrr"] Nov 01 22:23:43.357672 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.357647 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010a5ac6_aeef_45ce_8c57_dcebf29427b1.slice/crio-937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3 WatchSource:0}: Error finding container 937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3: Status 404 returned error can't find the container with id 937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3 Nov 01 22:23:43.400407 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.400370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:43.401038 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.401013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:43.403304 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.403284 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:43.403639 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.403625 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:43.414255 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.414230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:43.414346 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.414240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:43.557481 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.557443 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:43.565824 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.565798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:43.588075 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.588052 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:43.596148 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.596112 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:43.844140 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.843808 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhgqk"] Nov 01 22:23:43.847177 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.847142 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9445ebf3_feba_48ea_8eb6_882fd83343ab.slice/crio-5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5 WatchSource:0}: Error finding container 5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5: Status 404 returned error can't find the container with id 5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5 Nov 01 22:23:43.883916 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:43.883871 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhdh2"] Nov 01 22:23:43.888216 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:43.888163 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c7b714_b4a8_40b1_a4ec_2c0a2117e21c.slice/crio-9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3 WatchSource:0}: Error finding container 9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3: Status 404 returned error can't find the container with id 9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3 Nov 01 22:23:44.160734 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.160653 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zwrr" event={"ID":"010a5ac6-aeef-45ce-8c57-dcebf29427b1","Type":"ContainerStarted","Data":"937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3"} Nov 01 22:23:44.162466 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.162430 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhdh2" event={"ID":"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c","Type":"ContainerStarted","Data":"9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3"} Nov 01 22:23:44.165285 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.165224 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" event={"ID":"a8039013-0ab2-45d6-9321-da3ceb7dc478","Type":"ContainerStarted","Data":"9a600d7fe84ff7277cc87be0e40ebd42bd4d14d36a287d54fd6d60cd19d6f09a"} Nov 01 22:23:44.166891 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.166844 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" event={"ID":"1681c152-d4a7-417f-a54e-a7a4b11a2575","Type":"ContainerStarted","Data":"97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b"} Nov 01 22:23:44.168631 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.168607 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vhgqk" event={"ID":"9445ebf3-feba-48ea-8eb6-882fd83343ab","Type":"ContainerStarted","Data":"5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5"} Nov 01 22:23:44.170643 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.170618 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" event={"ID":"a0650f02-2be4-4fad-bd28-2fa18f86c6af","Type":"ContainerStarted","Data":"c15f781522671737af71c2174127ccc30ce8a147c17fcc6d52ae3bd34009b0dd"} Nov 01 22:23:44.170745 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.170663 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" event={"ID":"a0650f02-2be4-4fad-bd28-2fa18f86c6af","Type":"ContainerStarted","Data":"4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b"} Nov 01 22:23:44.192820 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:44.192425 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" podStartSLOduration=473.192406929 podStartE2EDuration="7m53.192406929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:44.190868378 +0000 UTC m=+66.987643837" watchObservedRunningTime="2026-11-01 22:23:44.192406929 +0000 UTC m=+66.989182395" Nov 01 22:23:45.021330 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:45.021235 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:45.024390 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:45.024165 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:45.176310 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:45.176237 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:45.177744 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:45.177712 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:46.189168 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.189123 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" event={"ID":"bc99826a-2d2c-41ae-b55a-e95475056ead","Type":"ContainerStarted","Data":"12f2ef2d02da16f01c2d696de1a6fd0c4455acd77d42a51360a339271d31ade1"} Nov 01 22:23:46.192493 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.192459 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhdh2" event={"ID":"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c","Type":"ContainerStarted","Data":"f3ebbdeec58697910a3614140536fed8dc69564ec8cdb6690cb7f37d4552071a"} Nov 01 22:23:46.192672 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.192518 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhdh2" event={"ID":"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c","Type":"ContainerStarted","Data":"4b3eb9390fdd7042091a7c01f70bbe5e359dd748f077641cf21152fb3ba049b5"} Nov 01 22:23:46.195948 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.195344 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" event={"ID":"a8039013-0ab2-45d6-9321-da3ceb7dc478","Type":"ContainerStarted","Data":"917f64714dbe1ebbd2746ff0722d3476ed1242e4e00bc75f65010f08abd4f678"} Nov 01 22:23:46.195948 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.195427 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" event={"ID":"a8039013-0ab2-45d6-9321-da3ceb7dc478","Type":"ContainerStarted","Data":"dacb0c6b1ad5488956a91f6c8250d21af20a55ed0b809d31bb59c15f660db7f9"} Nov 01 22:23:46.198700 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.198666 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" event={"ID":"1681c152-d4a7-417f-a54e-a7a4b11a2575","Type":"ContainerStarted","Data":"afe3fd97c47a50ed25afead5489b88d25c9a47f007a5847a9e339af6c90ad709"} Nov 01 22:23:46.209579 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.209518 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" podStartSLOduration=475.716881664 podStartE2EDuration="7m58.209500885s" totalImagesPullingTime="2.492619221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:46.208017046 +0000 UTC m=+69.004792502" watchObservedRunningTime="2026-11-01 22:23:46.209500885 +0000 UTC m=+69.006276349" Nov 01 22:23:46.250535 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.250482 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jhdh2" podStartSLOduration=66.475301598 podStartE2EDuration="1m8.250465621s" totalImagesPullingTime="1.775164023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:46.231064005 +0000 UTC m=+69.027839469" watchObservedRunningTime="2026-11-01 22:23:46.250465621 +0000 UTC m=+69.047241085" Nov 01 22:23:46.250920 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.250887 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" podStartSLOduration=473.63070003 podStartE2EDuration="7m56.250877354s" totalImagesPullingTime="2.620177324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:46.24925255 +0000 UTC m=+69.046028014" watchObservedRunningTime="2026-11-01 22:23:46.250877354 +0000 UTC m=+69.047652819" Nov 01 22:23:46.266229 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.266182 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" podStartSLOduration=442.817377444 podStartE2EDuration="7m25.266172847s" totalImagesPullingTime="2.448795403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:46.266044561 +0000 UTC m=+69.062820024" watchObservedRunningTime="2026-11-01 22:23:46.266172847 +0000 UTC m=+69.062948311" Nov 01 22:23:46.623446 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.623406 2573 patch_prober.go:29] interesting pod/image-registry-5bb5b5746c-v88s9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 22:23:46.623667 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:46.623464 2573 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:23:47.206565 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:47.206530 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zwrr" event={"ID":"010a5ac6-aeef-45ce-8c57-dcebf29427b1","Type":"ContainerStarted","Data":"f7b8b30b3b656a2a7400a9f2cb031cacc2538d320f9c7bab6637fc1daf28ace0"} Nov 01 22:23:47.206973 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:47.206615 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zwrr" event={"ID":"010a5ac6-aeef-45ce-8c57-dcebf29427b1","Type":"ContainerStarted","Data":"45bb79f52c960b488c2561fc9fa3f79d170112c036b060e8be4c0119bd916569"} Nov 01 22:23:47.207986 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:47.207958 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:47.222993 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:47.222940 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8zwrr" podStartSLOduration=34.281331632 podStartE2EDuration="37.222923144s" totalImagesPullingTime="2.941591512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:47.221770846 +0000 UTC m=+70.018546313" watchObservedRunningTime="2026-11-01 22:23:47.222923144 +0000 UTC m=+70.019698610" Nov 01 22:23:48.113405 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.113366 2573 patch_prober.go:29] interesting pod/image-registry-5bb5b5746c-v88s9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 22:23:48.113600 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.113448 2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:23:48.662317 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.662282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:48.665156 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.665131 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:48.857512 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.857481 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:48.865351 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:48.865322 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:49.180783 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:49.180753 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jgr7"] Nov 01 22:23:49.183743 ip-10-0-141-223 kubenswrapper[2573]: W1101 22:23:49.183707 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63988932_7675_4400_beb1_610e2ecfde16.slice/crio-33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3 WatchSource:0}: Error finding container 33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3: Status 404 returned error can't find the container with id 33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3 Nov 01 22:23:49.214632 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:49.214561 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jgr7" event={"ID":"63988932-7675-4400-beb1-610e2ecfde16","Type":"ContainerStarted","Data":"33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3"} Nov 01 22:23:50.071409 ip-10-0-141-223 kubenswrapper[2573]: I1101 22:23:50.071378 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:50.141184 ip-10-0-141-223 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 22:23:50.156331 ip-10-0-141-223 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 22:23:50.156507 ip-10-0-141-223 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 22:23:50.156731 ip-10-0-141-223 systemd[1]: kubelet.service: Consumed 4.453s CPU time. Nov 01 22:23:50.169483 ip-10-0-141-223 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:23:50.261994 ip-10-0-141-223 kubenswrapper[5778]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:50.261994 ip-10-0-141-223 kubenswrapper[5778]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:23:50.261994 ip-10-0-141-223 kubenswrapper[5778]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:50.261994 ip-10-0-141-223 kubenswrapper[5778]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:50.264204 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264121 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.264204 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264203 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.264204 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264207 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264211 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264214 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264216 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264219 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264222 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264225 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264227 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264229 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264232 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264236 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264238 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264246 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264249 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264252 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264254 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264257 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264260 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264262 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.264316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264265 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264267 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264270 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264272 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264275 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264277 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264280 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264282 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264285 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264287 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264289 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264292 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264294 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264297 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264301 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264305 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264308 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264311 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264314 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264317 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.264837 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264319 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264322 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264325 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264327 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264331 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264334 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264336 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264339 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264342 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264344 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264347 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264349 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264354 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264356 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264359 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264362 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264364 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264367 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264369 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264372 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.265362 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264374 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264376 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264379 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264381 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264405 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264408 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264411 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264414 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264417 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264420 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264422 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264425 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264427 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264430 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264432 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264435 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264437 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264446 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264449 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264452 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.265982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264455 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264457 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264460 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264464 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264467 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264470 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264473 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264476 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264479 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264482 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264485 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264487 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264490 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264492 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264495 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264497 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264501 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264503 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264506 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.266514 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264508 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264511 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264513 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264516 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264518 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.264520 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265283 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265290 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265292 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265295 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265298 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265301 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265304 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265307 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265309 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265312 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265315 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265317 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265320 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265323 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.267052 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265326 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265329 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265332 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265334 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265337 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265339 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265342 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265344 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265347 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265349 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265352 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265354 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265357 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265359 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265362 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265365 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265367 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265370 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265372 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.267567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265375 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265378 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265380 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265382 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265385 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265389 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265391 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265394 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265397 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265399 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265402 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265404 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265406 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265409 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265411 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265414 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265416 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265418 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265421 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.268145 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265424 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265426 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265428 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265431 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265433 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265436 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265438 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265441 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265443 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265448 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265450 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265453 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265455 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265458 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265460 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265463 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265465 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265468 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265470 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265473 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.268640 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265475 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265478 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265481 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265483 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265486 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265488 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265491 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265493 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265495 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265498 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265500 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265503 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265505 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265508 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265510 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265513 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265515 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265519 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265523 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.269185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265526 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265528 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265532 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265535 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265538 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265540 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265543 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265545 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265550 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265553 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265556 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265559 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265562 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265565 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.265567 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266118 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266126 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266129 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266132 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.269668 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266135 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266137 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266140 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266142 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266145 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266147 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266150 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266153 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266155 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266158 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266160 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266163 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266165 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266168 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266170 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266173 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266175 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266177 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266180 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266183 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.270174 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266185 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266188 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266190 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266192 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266195 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266198 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266202 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266204 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266208 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266212 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266215 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266218 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266220 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266222 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266225 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266227 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266230 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266232 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266234 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.270765 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266237 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266242 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266245 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266247 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266250 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266253 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266255 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266258 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266260 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266263 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266266 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266268 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266271 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266274 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266276 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266279 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266281 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266284 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266288 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.271611 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266290 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266293 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266296 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266298 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266301 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266303 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266305 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266308 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266310 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266313 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266315 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266318 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266320 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266323 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266328 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266331 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266334 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266337 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266339 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.272412 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266342 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266344 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266347 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266350 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266353 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266355 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266358 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266360 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266363 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266365 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266368 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266370 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266374 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266376 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266379 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266382 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266384 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266387 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266393 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266397 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.272978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266401 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266405 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266408 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266412 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.266415 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266534 5778 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266554 5778 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266564 5778 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266571 5778 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266577 5778 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266582 5778 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266589 5778 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266595 5778 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266599 5778 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266603 5778 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266608 5778 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266613 5778 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266617 5778 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266621 5778 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266625 5778 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266629 5778 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266633 5778 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266638 5778 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:23:50.273624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266649 5778 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266653 5778 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266657 5778 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266662 5778 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266666 5778 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266677 5778 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266681 5778 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266686 5778 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266690 5778 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266695 5778 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266699 5778 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266703 5778 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266708 5778 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266713 5778 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266720 5778 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266725 5778 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266734 5778 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266738 5778 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266743 5778 flags.go:64] FLAG: --enable-server="true" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266748 5778 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266754 5778 flags.go:64] FLAG: --event-burst="100" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266759 5778 flags.go:64] FLAG: --event-qps="50" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266763 5778 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266766 5778 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:23:50.274267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266769 5778 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266773 5778 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266776 5778 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266779 5778 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266782 5778 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266785 5778 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266788 5778 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266791 5778 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266794 5778 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266797 5778 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266800 5778 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266803 5778 flags.go:64] FLAG: --feature-gates="" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266807 5778 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266810 5778 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266820 5778 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266824 5778 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266827 5778 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266830 5778 flags.go:64] FLAG: --help="false" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266833 5778 flags.go:64] FLAG: --hostname-override="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266836 5778 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266839 5778 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266842 5778 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266845 5778 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:23:50.274901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266868 5778 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266874 5778 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266878 5778 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266883 5778 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266886 5778 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266889 5778 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266892 5778 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266895 5778 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266898 5778 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266901 5778 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266904 5778 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266907 5778 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266910 5778 flags.go:64] FLAG: --lock-file="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266913 5778 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266916 5778 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266919 5778 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266925 5778 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266927 5778 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266939 5778 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266943 5778 flags.go:64] FLAG: --logging-format="text" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266945 5778 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266949 5778 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266952 5778 flags.go:64] FLAG: --manifest-url="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266954 5778 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266965 5778 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:23:50.275784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266968 5778 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266972 5778 flags.go:64] FLAG: --max-pods="110" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266975 5778 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266978 5778 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266980 5778 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266983 5778 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266986 5778 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266989 5778 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.266993 5778 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267001 5778 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267007 5778 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267010 5778 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267013 5778 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267016 5778 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267019 5778 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267022 5778 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267024 5778 flags.go:64] FLAG: --port="10250" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267027 5778 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267030 5778 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05fb184765a746cb2" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267033 5778 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267036 5778 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267039 5778 flags.go:64] FLAG: --register-node="true" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267042 5778 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267046 5778 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267049 5778 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267051 5778 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:23:50.276770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267054 5778 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267058 5778 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267061 5778 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267064 5778 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267067 5778 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267070 5778 flags.go:64] FLAG: --runonce="false" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267079 5778 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267082 5778 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267085 5778 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267088 5778 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267091 5778 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267094 5778 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267097 5778 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267100 5778 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267104 5778 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267106 5778 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267111 5778 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267114 5778 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267117 5778 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267120 5778 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267122 5778 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267128 5778 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267131 5778 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267134 5778 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267141 5778 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:23:50.277981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267144 5778 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267147 5778 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267150 5778 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267153 5778 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267156 5778 flags.go:64] FLAG: --v="2" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267164 5778 flags.go:64] FLAG: --version="false" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267168 5778 flags.go:64] FLAG: --vmodule="" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267172 5778 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267176 5778 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267357 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267361 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267363 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267366 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267368 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267376 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267379 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267382 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267385 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267387 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267390 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267392 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267398 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.279011 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267400 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267404 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267407 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267410 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267412 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267414 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267417 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267419 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267422 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267424 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267427 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267429 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267432 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267434 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267437 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267439 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267441 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267444 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267446 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.279946 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267449 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267451 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267454 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267457 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267459 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267461 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267470 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267473 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267475 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267478 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267482 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267485 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267489 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267492 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267495 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267498 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267500 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267503 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267506 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.280544 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267508 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267511 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267513 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267516 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267518 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267520 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267523 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267525 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267528 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267530 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267533 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267535 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267538 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267540 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267542 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267545 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267547 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267549 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267552 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.281203 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267554 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267562 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267565 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267567 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267570 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267572 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267576 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267579 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267583 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267585 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267588 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267590 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267593 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267596 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267598 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267600 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267603 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267605 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267608 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267610 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.281679 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267612 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267615 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267617 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267620 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267622 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267624 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267627 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267629 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267632 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267634 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267638 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267641 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267644 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267646 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267654 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267657 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.282214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.267683 5778 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267726 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267730 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267737 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267741 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267744 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267746 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267749 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267752 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267754 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267758 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267761 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267764 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267766 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267769 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267771 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267774 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267776 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267779 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267781 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.282706 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267784 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267786 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267789 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267791 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267794 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267796 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267798 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267801 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267803 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267806 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267808 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267816 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267819 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267821 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267824 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267828 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267831 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267833 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267836 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267838 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.283410 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267841 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267844 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267861 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267864 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267866 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267869 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267871 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267874 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267876 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267879 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267882 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267884 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267887 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267889 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267892 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267894 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267897 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267899 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267903 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.283993 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267907 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267910 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267913 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267915 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267918 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267926 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267929 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267931 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267935 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267939 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267941 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267944 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267946 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267949 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267951 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267954 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267956 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267959 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267961 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267963 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.284478 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267966 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267969 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267971 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267974 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267976 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267979 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267981 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267984 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.267986 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268001 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268044 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268049 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268052 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268054 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268057 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268059 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268062 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268064 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268068 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.285060 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268071 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268075 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268078 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268082 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268085 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268088 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268090 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268093 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.268108 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.273808 5778 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.273826 5778 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.273866 5778 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.273874 5778 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273936 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273942 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273945 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273948 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273950 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273953 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.285535 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273956 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273958 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273961 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273963 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273966 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273969 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273972 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273974 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273976 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273982 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273986 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273989 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273992 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273994 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.273998 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274001 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274003 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274006 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274009 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274012 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.286048 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274015 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274018 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274020 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274024 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274049 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274054 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274058 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274063 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274067 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274071 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274074 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274077 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274079 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274082 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274085 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274087 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274090 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274092 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274095 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274097 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.286581 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274101 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274103 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274105 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274108 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274111 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274113 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274116 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274119 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274122 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274124 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274127 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274129 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274132 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274134 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274136 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274139 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274142 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274145 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274147 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274150 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.287098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274153 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274155 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274158 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274160 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274162 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274165 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274167 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274170 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274173 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274175 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274178 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274180 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274183 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274187 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274190 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274192 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274195 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274197 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274200 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.287681 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274202 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274205 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274207 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274209 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274212 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274215 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274217 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274220 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274222 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274225 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274227 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274230 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274235 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274239 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274242 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274245 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274248 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274251 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274253 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.288206 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274257 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274262 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.274302 5778 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274520 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274530 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274534 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274537 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274540 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274545 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274548 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274550 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274553 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274556 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274559 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274561 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274564 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.288677 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274567 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274570 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274572 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274575 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274578 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274581 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274584 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274586 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274589 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274591 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274594 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274596 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274599 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274601 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274604 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274606 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274609 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274611 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274614 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.289146 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274616 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274619 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274622 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274624 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274627 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274629 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274632 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274635 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274638 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274641 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274644 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274646 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274649 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274651 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274654 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274657 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274659 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274661 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274664 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.289634 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274666 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274669 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274671 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274674 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274676 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274679 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274682 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274684 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274686 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274689 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274691 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274694 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274696 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274698 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274701 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274704 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274706 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274709 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274711 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274713 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.290147 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274716 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274719 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274721 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274725 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274729 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274731 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274734 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274736 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274739 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274742 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274744 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274747 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274749 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274751 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274754 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274756 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274759 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274761 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274763 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274766 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.290903 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274768 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274771 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274773 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274776 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274778 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274781 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274785 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274788 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274791 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274794 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274796 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274799 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274801 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274804 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274806 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.291599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.274834 5778 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274884 5778 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274889 5778 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274892 5778 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274895 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274898 5778 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274901 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274903 5778 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274908 5778 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274912 5778 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274915 5778 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274918 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274921 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274923 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274926 5778 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274928 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274931 5778 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274933 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274936 5778 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:50.292242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274938 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274941 5778 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274944 5778 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274946 5778 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274949 5778 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274951 5778 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274953 5778 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274956 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274958 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274960 5778 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274963 5778 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274965 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274968 5778 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274971 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274974 5778 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274976 5778 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274979 5778 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274981 5778 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274983 5778 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274986 5778 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:50.293160 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274989 5778 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274991 5778 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274994 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274996 5778 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.274999 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275001 5778 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275003 5778 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275006 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275008 5778 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275011 5778 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275013 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275016 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275018 5778 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275020 5778 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275023 5778 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275025 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275028 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275030 5778 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275032 5778 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:50.294088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275035 5778 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275038 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275040 5778 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275043 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275045 5778 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275048 5778 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275051 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275056 5778 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275059 5778 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275061 5778 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275064 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275066 5778 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275069 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275072 5778 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275074 5778 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275077 5778 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275079 5778 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275082 5778 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275085 5778 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275088 5778 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:50.294975 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275090 5778 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275092 5778 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275095 5778 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275098 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275100 5778 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275103 5778 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275106 5778 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275108 5778 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275110 5778 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275113 5778 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275115 5778 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275118 5778 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275120 5778 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275124 5778 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275127 5778 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275130 5778 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275133 5778 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275135 5778 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275138 5778 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275144 5778 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:50.295922 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275146 5778 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275149 5778 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275152 5778 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275154 5778 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275157 5778 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275159 5778 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275162 5778 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275165 5778 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:50.275168 5778 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.275229 5778 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276295 5778 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276368 5778 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276577 5778 server.go:1014] "Starting client certificate rotation" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276726 5778 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276780 5778 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:37 +0000 UTC" deadline="2028-08-12 20:33:14.642012964 +0000 UTC" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.276808 5778 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15598h9m24.365206724s" Nov 01 22:23:50.296981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.277092 5778 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:50.297747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.278349 5778 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:50.297747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.280472 5778 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:23:50.297747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.282445 5778 log.go:25] "Validated CRI v1 image API" Nov 01 22:23:50.297747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.283336 5778 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:23:50.297747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.285772 5778 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 84325919-1e48-4972-9222-efa0f7d1624e:/dev/nvme0n1p3 93ef4de6-8423-4fe1-9302-603ee9e9d2dd:/dev/nvme0n1p4] Nov 01 22:23:50.298024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.285799 5778 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35e9a5ed2a4ec2855d5565ddfce2c6e93ca5ff316e67c7e68fc62000d6b249d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35e9a5ed2a4ec2855d5565ddfce2c6e93ca5ff316e67c7e68fc62000d6b249d8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5064614125a8f7d8f78041e5087a6f44cd66abaf4166821462e8366885397acb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5064614125a8f7d8f78041e5087a6f44cd66abaf4166821462e8366885397acb/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a600d7fe84ff7277cc87be0e40ebd42bd4d14d36a287d54fd6d60cd19d6f09a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a600d7fe84ff7277cc87be0e40ebd42bd4d14d36a287d54fd6d60cd19d6f09a/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~projected/kube-api-access-v46dx:{mountpoint:/var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~projected/kube-api-access-v46dx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06fbbe23-1f70-4c1e-b82f-5fa72a4166db/volumes/kubernetes.io~projected/kube-api-access-mkpx7:{mountpoint:/var/lib/kubelet/pods/06fbbe23-1f70-4c1e-b82f-5fa72a4166db/volumes/kubernetes.io~projected/kube-api-access-mkpx7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~projected/kube-api-access-92dfm:{mountpoint:/var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~projected/kube-api-access-92dfm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1681c152-d4a7-417f-a54e-a7a4b11a2575/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/1681c152-d4a7-417f-a54e-a7a4b11a2575/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/kube-api-access-wnc6h:{mountpoint:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/kube-api-access-wnc6h major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/registry-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29c2c4f7-79ec-444c-be78-ce43edf6850c/volumes/kubernetes.io~projected/kube-api-access-l69zf:{mountpoint:/var/lib/kubelet/pods/29c2c4f7-79ec-444c-be78-ce43edf6850c/volumes/kubernetes.io~projected/kube-api-access-l69zf major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ad608dd-a3a4-47cb-9702-4cbef303f2f7/volumes/kubernetes.io~projected/kube-api-access-ltb82:{mountpoint:/var/lib/kubelet/pods/3ad608dd-a3a4-47cb-9702-4cbef303f2f7/volumes/kubernetes.io~projected/kube-api-access-ltb82 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41572fd2-3250-4c36-a7be-8c8f733c39c3/volumes/kubernetes.io~projected/kube-api-access-jq5wg:{mountpoint:/var/lib/kubelet/pods/41572fd2-3250-4c36-a7be-8c8f733c39c3/volumes/kubernetes.io~projected/kube-api-access-jq5wg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~projected/kube-api-access-7zwbk:{mountpoint:/var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~projected/kube-api-access-7zwbk major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~projected/kube-api-access-p5qtk:{mountpoint:/var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~projected/kube-api-access-p5qtk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~projected/kube-api-access-k6fdk:{mountpoint:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~projected/kube-api-access-k6fdk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62bfdd4e-d511-4e4d-a225-17787716d664/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/62bfdd4e-d511-4e4d-a225-17787716d664/volumes/kubernetes.io~secret/agent-certs major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~projected/kube-api-access-ksncs:{mountpoint:/var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~projected/kube-api-access-ksncs major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65225fe4-74e2-4b06-9169-7dce163b2fec/volumes/kubernetes.io~projected/kube-api-access-p689n:{mountpoint:/var/lib/kubelet/pods/65225fe4-74e2-4b06-9169-7dce163b2fec/volumes/kubernetes.io~projected/kube-api-access-p689n major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~projected/kube-api-access-4qmz4:{mountpoint:/var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~projected/kube-api-access-4qmz4 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~projected/kube-api-access-56b7p:{mountpoint:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~projected/kube-api-access-56b7p major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~projected/kube-api-access-lg9mz:{mountpoint:/var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~projected/kube-api-access-lg9mz major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9445ebf3-feba-48ea-8eb6-882fd83343ab/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9445ebf3-feba-48ea-8eb6-882fd83343ab/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~projected/kube-api-access-gf2hg:{mountpoint:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~projected/kube-api-access-gf2hg major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/default-certificate major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/metrics-certs major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/stats-auth major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6acb24b-4146-452f-b222-008b883c77d0/volumes/kubernetes.io~projected/kube-api-access-vjk4g:{mountpoint:/var/lib/kubelet/pods/a6acb24b-4146-452f-b222-008b883c77d0/volumes/kubernetes.io~projected/kube-api-access-vjk4g major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~projected/kube-api-access-qj7jk:{mountpoint:/var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~projected/kube-api-access-qj7jk major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb90f847-aeda-4739-a6ac-33626b1e17a8/volumes/kubernetes.io~projected/kube-api-access-qd5pv:{mountpoint:/var/lib/kubelet/pods/bb90f847-aeda-4739-a6ac-33626b1e17a8/volumes/kubernetes.io~projected/kube-api-access-qd5pv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~projected/kube-api-access-tqrqx:{mountpoint:/var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~projected/kube-api-access-tqrqx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c545d6bc-16aa-4c0d-b30e-2ac42f675347/volumes/kubernetes.io~projected/kube-api-access-krx4b:{mountpoint:/var/lib/kubelet/pods/c545d6bc-16aa-4c0d-b30e-2ac42f675347/volumes/kubernetes.io~projected/kube-api-access-krx4b major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2372e60-3b4e-407f-b0ed-6799496e9dd7/volumes/kubernetes.io~projected/kube-api-access-lt58s:{mountpoint:/var/lib/kubelet/pods/d2372e60-3b4e-407f-b0ed-6799496e9dd7/volumes/kubernetes.io~projected/kube-api-access-lt58s major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3183d19-6697-470b-b5ff-cad0136248fa/volumes/kubernetes.io~projected/kube-api-access-4dngg:{mountpoint:/var/lib/kubelet/pods/d3183d19-6697-470b-b5ff-cad0136248fa/volumes/kubernetes.io~projected/kube-api-access-4dngg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~projected/kube-api-access-wjmds:{mountpoint:/var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~projected/kube-api-access-wjmds major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ca1661-6040-419a-9037-1241908c41fa/volumes/kubernetes.io~projected/kube-api-access-4jpfb:{mountpoint:/var/lib/kubelet/pods/d5ca1661-6040-419a-9037-1241908c41fa/volumes/kubernetes.io~projected/kube-api-access-4jpfb major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/kube-api-access-sstm7:{mountpoint:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/kube-api-access-sstm7 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/registry-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e98acdf9-450d-46a8-b3e8-faae9509a855/volumes/kubernetes.io~projected/kube-api-access-tvh6m:{mountpoint:/var/lib/kubelet/pods/e98acdf9-450d-46a8-b3e8-faae9509a855/volumes/kubernetes.io~projected/kube-api-access-tvh6m major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5b9510-e846-4249-911e-6f04e319853f/volumes/kubernetes.io~projected/kube-api-access-67zck:{mountpoint:/var/lib/kubelet/pods/fb5b9510-e846-4249-911e-6f04e319853f/volumes/kubernetes.io~projected/kube-api-access-67zck major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/71fa6014f5305c4eb47fb8154724657d3946c70297df8dedf7e381536902e1c3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d18fed8e09bbecbdbb7112e1af8ca1d656cb64c357c5bd206c54ee15f5143e34/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/77100e5fd5b30db638f50e72f11705a6e995a3f1378e1c6ef6cffb12f605afb5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/78e4af75217f18650245c8e22d47e1c75c47d8886da3b558bb4cc7a8c6e71cd0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d434831844e4c29969cb963ea5dd637944fd83ba710346fe6941c7e7ee7ba7de/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1b38c20bda9ddfe7ae8268e6b47f9bf26d491b8c0f2def3d8f6a62024175959d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/768036922cb4e867d0ca083b2523fae680a2766dfc77ce47c3920ee3f7f2c1d3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2459d203a1228f993d87f67da4bb4b6aaf0366680c5f15c281836bfda392b2d1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/18d98b7acfedc35273e6bf509068d04e168b18a9c014f60b45e46bdd1b32f630/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/0c1b7eb72f86b2690b019b60d64c377ecad1ef251341291f8c8a0bf4a02a3686/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/acbeaff53a477e23ea717031fd9f16cf5e10c0a2f4628a32dc4aa63e5221007c/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/5a0a9c0293142c02773ba7b321305d92a9c1f6fde89aef70633631dcd67cea51/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a6124a8141e1fc1f5fd62ae0d45d9c470e2db0b5dec84b3b187577178da79e64/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/214fd000f46b7bdf8405cd904631df31ebea7946030e10dd22340cbb6a1947bc/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/952ef1bb184eb6d4e0e968c20ae55b6ff30e7485ae661be82c90481dbc43979d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bed86d1c091fe506f89c01b73fea5a8179354d00fc19a1f0338124a0c430300f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/962e0a6ce6fb2a531149d214016d9590ea7216d9445b53742968d0d2bd0d76d0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1f611313beca5a23759777181168155ac6352216b5bce0c3968bf9c82a309ed5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c81736c81c11fa42663d29b7d1b621e40e09f0e0db13a1cdcb885739a2694ad0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/79cca952ae2acb723a38b1ea714630df15db9d784f39d4c8e197cd15f790a711/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f99ad46eda359bbe2dd42520da1915ef7a80eb4c9ef3189a0cdb9844948b2aa2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/530ea73a788c69972ae4a07e34c7a32d75fc9593a41f40b97555fcfc267edb90/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/98b79db9f030934a145ae7cb4a134138886f8d6b85cd25b29c1e60b85af27d94/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ececcf9dfc14318d62386d7ad19cc1d5cbbb07dec8ce988385ccc8cbda7efb89/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c4f99e9fac69e4e2ab70133054330af38d90f939cf2fcb9eed35c5891f5c38cb/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/9e2429e9fcc4f4f483a39aa52f62b54460bb425b692d50e250e047bc978592b2/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/99b353abda61ab8ee2c30db0d465cd76e675535b91d05eb76f232c0e7e623425/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/008c742f5b59e0fbb6a55230e016198836240fc4c44e4e5ea33a861169788a08/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/d9abc0561f2fbb24e251ffa3d29a18f496138e8f4de69b7a0bb6267163c2363f/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/de54d72ef3240296b79c99b6eacbae0b99fb31fc79461d22e19135767592c668/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/a422157cf37fad20090ef6f7f29838e0cb5c271565533d8b19fde54a7b7d841e/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/d8edd88457ee21d4735946c6e48b12e527df7d2a175137dbc92779780459afcc/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/81a7db462ce8a362038381035e724bd365cf96389343395d333ffe830ee946c4/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/b797e00d87b7a3af537916ba01686ba61cb74193eb42e10bba00aa9b512df124/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/5ebc03c3777784f07eb389bed6de3b1e73e9233af3beebdc3e88c3a56aec0c9f/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/296706523e1c1c71a3e3f4973a62894c13ee98e0d432c08496406927e68631c7/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/b1502427268accdd051129e045c27e4e4b7214e04f18881ce91c7c5c7b8389a3/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0fc2d9b6960193cbb198865b1de7efda5fe672191237d49962c831975742a274/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/cfe8db8ceda112aa3016f31337af6baed3275936cfc699980f9604369f86f23c/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/d51fea2d3fb00cfaa982a95bd77df778ac7b63ea29bb90a17202bf3e7b30c208/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/ff657857e3c8fec491ee7c1f8557d2728c8edf9b29e65a440767d1816f842c0d/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/f96f4d32491247c760a58b6eca00b427c6013dc2f0daaa8057210b10b6203487/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/8adc8190af0f3c9acd182d3a16cf621b6ff821d4396734b4cea876a63ba0af84/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/a4acb8c34b897fe8e4d823e5f45785779edf626cd78d1851266fbf408e6333a2/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/b364b59631258f5e1862990e766f3532733a079f4cbeaa84605267789c1f0006/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/e5a27cadf5babdf237dcf5005bb318b7bbfa2038b0fa08f57834aa160fd0ebc1/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/1ca1a1052e27401c160fead1d00e4e408299a47b15cf160a4081f181706017ed/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/b86a0af7f13148519a6cdaa87b7d666503f249ff41e2ef047cd0ddce6c65fc90/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/2fd6be7e2bd22fc2f99bd7a8529839e4ec7665f4a073d16d191eb956fa14aa80/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/0d00a49f824279d82091ecd31c526aa2096dbe18629184f15d2bc564189dc759/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/71ca39b9dc22bb9a6fb811d74fecc31231dcec5ddc9433e4abbb7f6b45504145/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/09f485c004ccad86fc8b00420ebc3b37380290fcf8280344bf66fdcc443709f6/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/2617d7ab8b6063a8cb7ba9e8d1a281019d5ee98dbfb350819da4e11ee2fdc0fd/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/08a0fbeb7eccc53b11ea2a1cd4591cc2dffa0310ea89572756214de2fc5bcc30/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/c09575dda8fe3847910121370efc00f41636122c1fc4734ac34bed2d8259f2d3/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/738a93c0766d18bc59d34898d0a845fa9ebfe4a5da039582fd67d6df28a266bf/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/a330892a050afef1dd027a0193a13c7bde18a40b5f9541a0a346a0b88a6d91b6/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/65a50f5b0f78a0b1533e4cbb3b070552a845db52af4c9ad77c9d5587df84acc0/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/79c0283cdf729d6bfc01703dab031a22c3b491398720b6194e9f989a961ea1e2/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/96c7846285fced0103ba9f7d0042746cd878ea73078d5f5d58018658ddf48eb6/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/3f5f630c2c146f6562edba5c304c79960d4b308b2b7dfbe19d4b701f9d8d0e4f/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/f326f3add05a3f08a7e04ed747992715cbc5a430a8278b8e95de24b93e25003c/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/c7d524dd725b75e6dbd054bdf12b739c0caa29fbf3af78da9f6b2852c6a549a2/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/035ddfeb49298587555f57e3690a130a7e6db11a7f713e61b04dea1dc42267e8/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/e960ecddd4ccd4b90e149a1d6f32747dea45e9f8b1cad7d3b3c9c5629859adba/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/cb940fe5c5d2b38bef93c65129ce618b0405a0678c925684252566da561cf237/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/8cf1bea0515080850ee0531c938146430ba86f47cbb5e815d9f9c459725133d3/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/b95fb86f411d9828e474436ec609d03e0dea0d0e6684226a0c3ec6acb552fe75/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b679942f4e5756b60f89c5bf5ed261d11f8352e795ed8b5e5180bba436b69ef9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a04427e578273f6348028a02a74ad52d8ee97317f516c7bf01ecabb22277a6a6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8f0d5a6e543462ac1d4485f7e8a7828b3e92a7cb54ed2e248b7b2e28ae784e12/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/75d9582235b4b5fe651308a2491d246df2eb5188ad39a7c05082e6067bf6ad55/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7bd861c323cf58bd852da4c276aecb4fa2c7db5cdac30f7199259ef5ee1976df/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/78da6e40ba2913c182c8c649e62bbd8ae894ddd0e999cba150a084c259f0746e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/304241e4381de78cb3e6c19fc5fdc0a2c2c3d11c6df824810d01fa0e1201caa7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9bea938283650d233ecc5085fe5605c4c3e042bc433a37efd39051a7ed92b42e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/bd53d93de5e2b80067c91034eda02b54314c61d2b5ff97542c3f016e47535f40/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/698aea5b6602639fd460b657d97a4e1d5ebc764d010c1b140b2d396ac8107a35/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 22:23:50.298671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.298026 5778 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:50.297181758 +0000 UTC m=+0.073168580 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093431 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e4ade4504eea73898a9accdf403d6 SystemUUID:ec2e4ade-4504-eea7-3898-a9accdf403d6 BootID:28c7fdc2-0259-4a40-9cbb-b05b671ff20b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74d2155006a7ad8f7a336ade541d43594792d5a4ee64db93d4dbe205c9539160/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~projected/kube-api-access-wjmds DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9445ebf3-feba-48ea-8eb6-882fd83343ab/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:409 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f1bdfd4af47fc358e9961dc4e456b6f29c4527a4d12fdf9a903a8ed81eceaf3/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ca1661-6040-419a-9037-1241908c41fa/volumes/kubernetes.io~projected/kube-api-access-4jpfb DeviceMajor:0 DeviceMinor:262 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~projected/kube-api-access-k6fdk DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~projected/kube-api-access-56b7p DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2907340b90a0ed138b14ec7d543904cca0bc3e133cdd0f167842ef87fb71c14/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cbd3528ccfba7809a9e169b471c4a82eb7c38433a66b4316fb1361c2f54d7d6/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/65225fe4-74e2-4b06-9169-7dce163b2fec/volumes/kubernetes.io~projected/kube-api-access-p689n DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a600d7fe84ff7277cc87be0e40ebd42bd4d14d36a287d54fd6d60cd19d6f09a/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d5d449db8a061ce5d26ab9c110911a8e767a720a5f7ba2ccfe402f2746c730b/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22203f5ef2a639117a20f610b6a1e842e367ce0c316fd5274d9c129a2ddf28b0/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5064614125a8f7d8f78041e5087a6f44cd66abaf4166821462e8366885397acb/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96ca3deeafa866662a19f54538ac0c23f1684b4f4550dd7985cbf04b38cf3e87/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88e4e6a5f25b627ffc2f0bdc24a59251fb35672953ec656be279b0374f527e7d/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b16a7c81ae2080e5ca5ad3da7e546ef031b22f9efe899a062d2f62fd895647b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d47d3a1e-edf7-4b20-b30c-96449721dbef/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/49f57d53-5511-4fc4-86b1-a4884a4d213a/volumes/kubernetes.io~projected/kube-api-access-7zwbk DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:375 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97a096e1c296cd6f0bf001e12d182c80eec16156d80e1dfbf6a670a21c382d5b/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81b411a3-fb23-4551-894a-958b2404c226/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a5628c025b5808cbce6fd92af9ee436fd57fdd5187110bae8263faa4ab847c7/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~projected/kube-api-access-tqrqx DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~projected/kube-api-access-ksncs DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:377 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2372e60-3b4e-407f-b0ed-6799496e9dd7/volumes/kubernetes.io~projected/kube-api-access-lt58s DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c413c83eec1f2b92a1fb50e9411536aa6cca2f9df5ffaefe53b22eba75259b9/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7c3638223ec04db216cd63a43bb1461aec9ee79e078dda523fd8c20b01f9610/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc99826a-2d2c-41ae-b55a-e95475056ead/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63988932-7675-4400-beb1-610e2ecfde16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:459 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d56e76ebb40cb1f32c355c35ddc29028e4f26176d0c90abcb978ed2572d4b2f2/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62bfdd4e-d511-4e4d-a225-17787716d664/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ad608dd-a3a4-47cb-9702-4cbef303f2f7/volumes/kubernetes.io~projected/kube-api-access-ltb82 DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29c2c4f7-79ec-444c-be78-ce43edf6850c/volumes/kubernetes.io~projected/kube-api-access-l69zf DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0650f02-2be4-4fad-bd28-2fa18f86c6af/volumes/kubernetes.io~projected/kube-api-access-gf2hg DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb5b9510-e846-4249-911e-6f04e319853f/volumes/kubernetes.io~projected/kube-api-access-67zck DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/kube-api-access-sstm7 DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec728cb36f9aa94427f43f5de17d45e2828d8dc5cc29d969364d57710c974757/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~projected/kube-api-access-lg9mz DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f2587e9eb33a913d3c49aed0e34ecc80354a309653b03fd96bc820edf7ffdd5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~projected/kube-api-access-92dfm DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb90f847-aeda-4739-a6ac-33626b1e17a8/volumes/kubernetes.io~projected/kube-api-access-qd5pv DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33a3ba0c19735eb65803f0091a31c9c257a416f5ec69a302f3fa08aac22659d3/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41572fd2-3250-4c36-a7be-8c8f733c39c3/volumes/kubernetes.io~projected/kube-api-access-jq5wg DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/400d6ec3edab7223d3ce202face7e73180557316857648ac91fe00aaddbee6b5/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35e9a5ed2a4ec2855d5565ddfce2c6e93ca5ff316e67c7e68fc62000d6b249d8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ab12498ebb715f2e058cd63facf3360f7ba0667e1dbeffce4542a80a1b04fd2/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/kube-api-access-wnc6h DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c081063fa75212d9ecb8923841cef97a5bfbdb963e4e8f0711a6f837c5be95b/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/06fbbe23-1f70-4c1e-b82f-5fa72a4166db/volumes/kubernetes.io~projected/kube-api-access-mkpx7 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c627c44-b90e-4477-ade0-1a9ce7aafb36/volumes/kubernetes.io~projected/kube-api-access-4qmz4 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e98acdf9-450d-46a8-b3e8-faae9509a855/volumes/kubernetes.io~projected/kube-api-access-tvh6m DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1681c152-d4a7-417f-a54e-a7a4b11a2575/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:369 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/937e8ab63409a49ad4541d8f65824ed02de8df0c988510075bdb285cbff2bbe3/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~projected/kube-api-access-p5qtk DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6acb24b-4146-452f-b222-008b883c77d0/volumes/kubernetes.io~projected/kube-api-access-vjk4g DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e707915-61a5-42f8-bb48-920bda324148/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd153e6a48819c009378b85a25a56384212938856c76110354c433cc73f86c93/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcb784edcf69230f8fba9d5789a1cad252509bf9673ee9d4b8af684ea64e842e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d3307fcf3a6226f52d531a3282b2142f19edf9f1a991939c54d5e7343a250e5/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/010a5ac6-aeef-45ce-8c57-dcebf29427b1/volumes/kubernetes.io~projected/kube-api-access-v46dx DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07772e605a33f9f66f061b57a9d92c7a8a1ceb77e49ecf8781f99e6ee4b2a1c8/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa5e010-5d31-46f0-bb5a-a952613af986/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d3183d19-6697-470b-b5ff-cad0136248fa/volumes/kubernetes.io~projected/kube-api-access-4dngg DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:402 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a8039013-0ab2-45d6-9321-da3ceb7dc478/volumes/kubernetes.io~projected/kube-api-access-qj7jk DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f84be26-280f-4289-a270-45b31fcd6adc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c545d6bc-16aa-4c0d-b30e-2ac42f675347/volumes/kubernetes.io~projected/kube-api-access-krx4b DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:038f1f74ca4041e MacAddress:ce:9a:b5:84:a3:d2 Speed:10000 Mtu:8901} {Name:07772e605a33f9f MacAddress:c2:b4:f6:c5:ec:25 Speed:10000 Mtu:8901} {Name:22203f5ef2a6391 MacAddress:de:54:4f:00:68:eb Speed:10000 Mtu:8901} {Name:33a3ba0c19735eb MacAddress:8a:a7:48:6d:cd:f4 Speed:10000 Mtu:8901} {Name:39ff92eb6f46973 MacAddress:0e:86:50:4e:d8:3d Speed:10000 Mtu:8901} {Name:400d6ec3edab722 MacAddress:be:e9:4b:39:24:22 Speed:10000 Mtu:8901} {Name:4c081063fa75212 MacAddress:36:2e:5f:b0:30:e3 Speed:10000 Mtu:8901} {Name:5064614125a8f7d MacAddress:ea:a8:4f:c3:e0:bf Speed:10000 Mtu:8901} {Name:5d3307fcf3a6226 MacAddress:ee:5c:ee:fe:bc:db Speed:10000 Mtu:8901} {Name:6b16a7c81ae2080 MacAddress:fa:25:4f:55:6b:b9 Speed:10000 Mtu:8901} {Name:88e4e6a5f25b627 MacAddress:2a:a0:d9:e4:38:30 Speed:10000 Mtu:8901} {Name:8a5628c025b5808 MacAddress:fe:c3:8b:c8:33:0e Speed:10000 Mtu:8901} {Name:8cbd3528ccfba78 MacAddress:fa:18:fe:c9:ba:fa Speed:10000 Mtu:8901} {Name:937e8ab63409a49 MacAddress:86:e0:dd:1d:f8:5d Speed:10000 Mtu:8901} {Name:97a096e1c296cd6 MacAddress:9e:fe:66:ed:c4:e2 Speed:10000 Mtu:8901} {Name:9a600d7fe84ff72 MacAddress:86:be:00:cf:bd:75 Speed:10000 Mtu:8901} {Name:9f1bdfd4af47fc3 MacAddress:02:a0:84:05:15:e9 Speed:10000 Mtu:8901} {Name:bcb784edcf69230 MacAddress:62:26:8b:3c:44:ab Speed:10000 Mtu:8901} {Name:bd153e6a48819c0 MacAddress:46:fc:ee:4e:38:a7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e8:13:7f:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:3f:db:7c:76:44 Speed:0 Mtu:8901} {Name:c2907340b90a0ed MacAddress:42:6f:4d:a0:6b:aa Speed:10000 Mtu:8901} {Name:d56e76ebb40cb1f MacAddress:be:20:69:b5:0c:e2 Speed:10000 Mtu:8901} {Name:d7c3638223ec04d MacAddress:f6:da:92:99:c3:55 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e8:13:7f:49 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:44:24:32:4d:d0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:ce:e6:9c:c5:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:23:50.299129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.298682 5778 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:23:50.299129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.298748 5778 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:23:50.299247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299140 5778 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:23:50.299380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299168 5778 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-223.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:23:50.299454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299384 5778 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:23:50.299454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299397 5778 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:23:50.299454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299411 5778 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:50.299454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299438 5778 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:50.299635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299553 5778 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:23:50.299782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299770 5778 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:23:50.299984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.299971 5778 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:23:50.300070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.300035 5778 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:23:50.300070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.300050 5778 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:23:50.300070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.300063 5778 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:23:50.300228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.300077 5778 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:23:50.300228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.300090 5778 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:23:50.302185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.302165 5778 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:50.302185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.302183 5778 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:50.302841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.302820 5778 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:23:50.314624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.314609 5778 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:50.314624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.314623 5778 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:50.314744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.314723 5778 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 22:23:50.314948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.314936 5778 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:23:50.314988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.314966 5778 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:23:50.315082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315072 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:23:50.315113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315087 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:23:50.315113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315094 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:23:50.315113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315099 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:23:50.315113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315107 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315117 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315127 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315134 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315140 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315146 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315155 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315163 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315188 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:23:50.315240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.315197 5778 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:23:50.319617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.318759 5778 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:50.321226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.320130 5778 server.go:1263] "Started kubelet" Nov 01 22:23:50.321226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.320603 5778 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:23:50.321226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.320791 5778 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:23:50.321226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.320910 5778 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:23:50.320677 ip-10-0-141-223 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:23:50.322074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.322053 5778 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:23:50.322793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.322772 5778 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:50.323006 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.322990 5778 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:23:50.326662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326645 5778 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:23:50.326744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326719 5778 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:23:50.326822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326785 5778 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:39 +0000 UTC" deadline="2028-08-07 03:35:19.70432208 +0000 UTC" Nov 01 22:23:50.327161 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326825 5778 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15461h11m29.377501019s" Nov 01 22:23:50.327161 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326883 5778 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:23:50.327161 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326892 5778 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:23:50.327161 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.326966 5778 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:23:50.327638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.327613 5778 factory.go:55] Registering systemd factory Nov 01 22:23:50.327716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.327641 5778 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:23:50.328659 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.328638 5778 factory.go:153] Registering CRI-O factory Nov 01 22:23:50.328752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.328662 5778 factory.go:223] Registration of the crio container factory successfully Nov 01 22:23:50.328752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.328737 5778 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:23:50.328882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.328759 5778 factory.go:103] Registering Raw factory Nov 01 22:23:50.328882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.328776 5778 manager.go:1199] Started watching for new ooms in manager Nov 01 22:23:50.329261 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.329245 5778 manager.go:319] Starting recovery of all containers Nov 01 22:23:50.330003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.329964 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e707915-61a5-42f8-bb48-920bda324148" volumeName="kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config" seLinuxMountContext="" Nov 01 22:23:50.330003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330004 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f57d53-5511-4fc4-86b1-a4884a4d213a" volumeName="kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330019 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa5e010-5d31-46f0-bb5a-a952613af986" volumeName="kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330033 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc99826a-2d2c-41ae-b55a-e95475056ead" volumeName="kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330047 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc99826a-2d2c-41ae-b55a-e95475056ead" volumeName="kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330058 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc99826a-2d2c-41ae-b55a-e95475056ead" volumeName="kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330072 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e707915-61a5-42f8-bb48-920bda324148" volumeName="kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330086 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330101 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f84be26-280f-4289-a270-45b31fcd6adc" volumeName="kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330114 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2372e60-3b4e-407f-b0ed-6799496e9dd7" volumeName="kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330127 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2372e60-3b4e-407f-b0ed-6799496e9dd7" volumeName="kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330139 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330153 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5b9510-e846-4249-911e-6f04e319853f" volumeName="kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330166 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330184 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63988932-7675-4400-beb1-610e2ecfde16" volumeName="kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 22:23:50.330200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330198 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63988932-7675-4400-beb1-610e2ecfde16" volumeName="kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330210 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0650f02-2be4-4fad-bd28-2fa18f86c6af" volumeName="kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330222 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb90f847-aeda-4739-a6ac-33626b1e17a8" volumeName="kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330237 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41572fd2-3250-4c36-a7be-8c8f733c39c3" volumeName="kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330249 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f57d53-5511-4fc4-86b1-a4884a4d213a" volumeName="kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330264 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" volumeName="kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330277 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c627c44-b90e-4477-ade0-1a9ce7aafb36" volumeName="kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330289 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c627c44-b90e-4477-ade0-1a9ce7aafb36" volumeName="kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330301 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b411a3-fb23-4551-894a-958b2404c226" volumeName="kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330333 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0650f02-2be4-4fad-bd28-2fa18f86c6af" volumeName="kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330347 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330362 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="010a5ac6-aeef-45ce-8c57-dcebf29427b1" volumeName="kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330376 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06fbbe23-1f70-4c1e-b82f-5fa72a4166db" volumeName="kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330410 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e707915-61a5-42f8-bb48-920bda324148" volumeName="kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330424 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57c7b714-b4a8-40b1-a4ec-2c0a2117e21c" volumeName="kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330455 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65225fe4-74e2-4b06-9169-7dce163b2fec" volumeName="kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330468 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b411a3-fb23-4551-894a-958b2404c226" volumeName="kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330481 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b411a3-fb23-4551-894a-958b2404c226" volumeName="kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330494 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa5e010-5d31-46f0-bb5a-a952613af986" volumeName="kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330507 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330520 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad608dd-a3a4-47cb-9702-4cbef303f2f7" volumeName="kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330533 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3183d19-6697-470b-b5ff-cad0136248fa" volumeName="kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330544 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330557 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330570 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f84be26-280f-4289-a270-45b31fcd6adc" volumeName="kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330583 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330596 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6acb24b-4146-452f-b222-008b883c77d0" volumeName="kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330609 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3183d19-6697-470b-b5ff-cad0136248fa" volumeName="kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330621 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3183d19-6697-470b-b5ff-cad0136248fa" volumeName="kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330656 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330667 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330681 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330493 5778 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330693 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1681c152-d4a7-417f-a54e-a7a4b11a2575" volumeName="kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330705 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41572fd2-3250-4c36-a7be-8c8f733c39c3" volumeName="kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330717 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63988932-7675-4400-beb1-610e2ecfde16" volumeName="kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330732 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa5e010-5d31-46f0-bb5a-a952613af986" volumeName="kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330744 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8039013-0ab2-45d6-9321-da3ceb7dc478" volumeName="kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330756 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb90f847-aeda-4739-a6ac-33626b1e17a8" volumeName="kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330770 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330782 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330796 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b411a3-fb23-4551-894a-958b2404c226" volumeName="kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330819 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8039013-0ab2-45d6-9321-da3ceb7dc478" volumeName="kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330831 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330844 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ca1661-6040-419a-9037-1241908c41fa" volumeName="kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb" seLinuxMountContext="" Nov 01 22:23:50.330823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330877 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330890 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="010a5ac6-aeef-45ce-8c57-dcebf29427b1" volumeName="kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330903 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f84be26-280f-4289-a270-45b31fcd6adc" volumeName="kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330916 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0650f02-2be4-4fad-bd28-2fa18f86c6af" volumeName="kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330927 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c545d6bc-16aa-4c0d-b30e-2ac42f675347" volumeName="kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330940 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3183d19-6697-470b-b5ff-cad0136248fa" volumeName="kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330951 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5b9510-e846-4249-911e-6f04e319853f" volumeName="kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330965 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330978 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63988932-7675-4400-beb1-610e2ecfde16" volumeName="kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.330990 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8039013-0ab2-45d6-9321-da3ceb7dc478" volumeName="kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331003 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62bfdd4e-d511-4e4d-a225-17787716d664" volumeName="kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331016 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62bfdd4e-d511-4e4d-a225-17787716d664" volumeName="kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331028 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d47d3a1e-edf7-4b20-b30c-96449721dbef" volumeName="kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331041 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331055 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c3582e-4821-483f-b90a-6175bd9e27be" volumeName="kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331067 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5b9510-e846-4249-911e-6f04e319853f" volumeName="kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331079 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1681c152-d4a7-417f-a54e-a7a4b11a2575" volumeName="kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331092 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b411a3-fb23-4551-894a-958b2404c226" volumeName="kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331104 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9445ebf3-feba-48ea-8eb6-882fd83343ab" volumeName="kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331115 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0650f02-2be4-4fad-bd28-2fa18f86c6af" volumeName="kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331127 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="010a5ac6-aeef-45ce-8c57-dcebf29427b1" volumeName="kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331139 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331153 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331164 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e98acdf9-450d-46a8-b3e8-faae9509a855" volumeName="kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331176 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="010a5ac6-aeef-45ce-8c57-dcebf29427b1" volumeName="kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331187 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24621246-44ab-4de5-a8a8-389f98b09d22" volumeName="kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331199 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29c2c4f7-79ec-444c-be78-ce43edf6850c" volumeName="kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331213 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c627c44-b90e-4477-ade0-1a9ce7aafb36" volumeName="kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331225 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa5e010-5d31-46f0-bb5a-a952613af986" volumeName="kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331236 5778 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0650f02-2be4-4fad-bd28-2fa18f86c6af" volumeName="kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate" seLinuxMountContext="" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331248 5778 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:23:50.332732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.331255 5778 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:23:50.338476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.337958 5778 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 22:23:50.346916 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.346724 5778 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:23:50.348502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.348380 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:50.351157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.351128 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:50.365672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.365504 5778 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:23:50.365672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.365564 5778 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:23:50.365933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.365912 5778 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:23:50.366051 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.365972 5778 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 22:23:50.369597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.369568 5778 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:50.382794 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382757 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="5e2c64c6620c7090af1e90024a4cac01d2371710f542b755fb2d5e987d1d824f" exitCode=0 Nov 01 22:23:50.382794 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382786 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="770f613f98c7bd252bd8aebd239bce12f1bc57f98d2753d72b1dc2a4b4d90d65" exitCode=0 Nov 01 22:23:50.383007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382795 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="dbb0f0b86811ed59edfacc6d2d25c8c4633ada32570db890f700ebdafff41b89" exitCode=0 Nov 01 22:23:50.383007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382819 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="e9a5469b517408c71dfb019e66f6e73fa9584bc400655d90083378b3273c21b4" exitCode=0 Nov 01 22:23:50.383007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382827 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="ac41287c52579b7b41b1bf0016036735a990815e30f39de075ccc907f53ed842" exitCode=0 Nov 01 22:23:50.383007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.382834 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3183d19-6697-470b-b5ff-cad0136248fa" containerID="35d300d0f77ddd82999994ee018402d0a4e86a21f5dfa202410bbf1aec63edb3" exitCode=0 Nov 01 22:23:50.396641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.396583 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:50.396641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.396609 5778 generic.go:356] "Generic (PLEG): container finished" podID="8aa5e010-5d31-46f0-bb5a-a952613af986" containerID="ce64ea81fa701c64de0c825f0c32f0027359c469925158a813149280849a3be7" exitCode=255 Nov 01 22:23:50.436827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.436792 5778 generic.go:356] "Generic (PLEG): container finished" podID="5b928c74164b2c9fe5433adca4981b0d" containerID="97af0e3a36d0578cb05642ecc0ee672dda39c212771385ae162b390c6819f716" exitCode=0 Nov 01 22:23:50.446280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.446260 5778 manager.go:324] Recovery completed Nov 01 22:23:50.466046 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.466017 5778 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470007 5778 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470027 5778 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470048 5778 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470227 5778 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470244 5778 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470268 5778 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470277 5778 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470286 5778 policy_none.go:50] "Start" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470322 5778 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470335 5778 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470557 5778 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470567 5778 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 22:23:50.470825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.470575 5778 policy_none.go:44] "Start" Nov 01 22:23:50.479445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.478922 5778 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:23:50.485291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.485274 5778 manager.go:341] "Starting Device Plugin manager" Nov 01 22:23:50.485400 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.485302 5778 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:23:50.485400 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.485312 5778 server.go:87] "Starting device plugin registration server" Nov 01 22:23:50.485618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.485598 5778 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:23:50.485749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.485616 5778 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:23:50.485898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.485838 5778 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:23:50.487365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.487332 5778 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:23:50.487455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.487374 5778 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:23:50.487455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.487379 5778 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:23:50.487455 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.487348 5778 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:23:50.487588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.487451 5778 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:50.487365794Z","UUID":"8a76d40f-7350-477c-97bc-9f31d492f744","Handler":null,"Name":"","Endpoint":""} Nov 01 22:23:50.488965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.488936 5778 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:23:50.488965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.488960 5778 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:23:50.586693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.586662 5778 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:23:50.588673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.588653 5778 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:23:50.588778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.588691 5778 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:23:50.588778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.588705 5778 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-223.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:23:50.588778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.588761 5778 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.602695 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.602676 5778 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.602772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.602742 5778 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.666313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.666244 5778 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal"] Nov 01 22:23:50.666937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.666905 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"608dfbc50dc552b9f5f883d23954a6ba","Type":"ContainerStarted","Data":"6e30a9906548e531ba31fc8246b9796408bec177e0ef6fe71d900c255dfcb057"} Nov 01 22:23:50.666989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.666945 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"608dfbc50dc552b9f5f883d23954a6ba","Type":"ContainerStarted","Data":"625ba8ffc2238db1370177c47b48d59d26a9a4e2073f5c1667af55eae157f97d"} Nov 01 22:23:50.667037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.667014 5778 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.667037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.667023 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerStarted","Data":"b5c607e8a20a65dfab367f5a131a061ce50bea51627393a73f012caba26d54ab"} Nov 01 22:23:50.667110 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.667037 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerDied","Data":"97af0e3a36d0578cb05642ecc0ee672dda39c212771385ae162b390c6819f716"} Nov 01 22:23:50.667110 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.667049 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" event={"ID":"5b928c74164b2c9fe5433adca4981b0d","Type":"ContainerStarted","Data":"d29d42cc9df966e1c77baae08dc879017e109569859121591c87bbdc381cda1d"} Nov 01 22:23:50.667110 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.667017 5778 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.680504 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.680485 5778 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:50.680624 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.680533 5778 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.681131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.681117 5778 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:50.681181 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:50.681169 5778 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.739525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.739490 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.739525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.739525 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.739739 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.739556 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840050 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840074 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840101 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840139 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840175 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608dfbc50dc552b9f5f883d23954a6ba-config\") pod \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"608dfbc50dc552b9f5f883d23954a6ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:50.840285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:50.840191 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b928c74164b2c9fe5433adca4981b0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" (UID: \"5b928c74164b2c9fe5433adca4981b0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:51.302293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.302262 5778 apiserver.go:52] "Watching apiserver" Nov 01 22:23:51.320667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.320637 5778 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:23:51.321339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.321316 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4","kube-system/global-pull-secret-syncer-vhgqk","openshift-dns/node-resolver-t5wh4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb","openshift-network-console/networking-console-plugin-86c478bb88-2rlrx","openshift-image-registry/node-ca-xpgg7","openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz","openshift-ingress/router-default-7b8b6447f-8bbxr","openshift-ovn-kubernetes/ovnkube-node-pj8xg","openshift-network-operator/iptables-alerter-54bgs","openshift-multus/multus-7jrrm","openshift-console-operator/console-operator-7cd56b8dbc-nbf7s","kube-system/konnectivity-agent-w5598","openshift-dns/dns-default-8zwrr","openshift-multus/multus-additional-cni-plugins-h74xz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc","openshift-cluster-node-tuning-operator/tuned-ghlmj","openshift-ingress-canary/ingress-canary-wn5v5","kube-system/kas-connection-checker-685895c7c9-wtzbw","openshift-image-registry/image-registry-5bb5b5746c-v88s9","openshift-image-registry/image-registry-7fbdb597f4-jpdzl","openshift-insights/insights-runtime-extractor-2jgr7","kube-system/kas-connection-checker-685895c7c9-jpxjw","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx","openshift-network-diagnostics/network-check-target-zvpsn","openshift-insights/insights-operator-7c4986d449-nkrwh","kube-system/kas-connection-checker-685895c7c9-5j2lt","openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg","openshift-multus/network-metrics-daemon-jhdh2","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7","openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls"] Nov 01 22:23:51.326351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.326324 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.326663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.326554 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.326798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.326697 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.326892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.326708 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.327013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.326995 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 22:23:51.327259 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.327237 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.328273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.328227 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 22:23:51.328566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.328544 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.328662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.328644 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 22:23:51.328898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.328877 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.328989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.328977 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 22:23:51.329989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.329961 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.330673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.330647 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.331774 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.331750 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.332597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.332568 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 22:23:51.333555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333517 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 22:23:51.333639 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333589 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 22:23:51.333639 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333628 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.333754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333645 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.333754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333729 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 22:23:51.333880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333845 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 22:23:51.334119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.334099 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.334280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.333872 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 22:23:51.334483 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.334451 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.335144 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.334642 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 22:23:51.335215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.335185 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 22:23:51.335413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.335390 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.335484 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.335424 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 22:23:51.335562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.335548 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336633 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336664 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336678 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336688 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336701 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336747 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.336878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.336827 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.338040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.338015 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 22:23:51.338250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.338230 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.338555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.338530 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 22:23:51.340003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.339975 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.342071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342034 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.342071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342070 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.342229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342090 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.342229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342115 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpx7\" (UniqueName: \"kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7\") pod \"kas-connection-checker-685895c7c9-jpxjw\" (UID: \"06fbbe23-1f70-4c1e-b82f-5fa72a4166db\") " pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:51.342229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342139 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69zf\" (UniqueName: \"kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf\") pod \"kas-connection-checker-685895c7c9-wtzbw\" (UID: \"29c2c4f7-79ec-444c-be78-ce43edf6850c\") " pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:51.342374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342228 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2372e60-3b4e-407f-b0ed-6799496e9dd7-tmp-dir\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.342374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342271 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.342374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342298 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.342374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342319 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e707915-61a5-42f8-bb48-920bda324148-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.342374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342356 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.342608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342383 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltb82\" (UniqueName: \"kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82\") pod \"volume-data-source-validator-58c9f5f46d-znttb\" (UID: \"3ad608dd-a3a4-47cb-9702-4cbef303f2f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:51.342608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342468 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.342608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342494 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9mz\" (UniqueName: \"kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.342608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342551 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.342608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.342845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342635 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92dfm\" (UniqueName: \"kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.342845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342668 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.342845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342786 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.342845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342818 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.342845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342842 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342895 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342789 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d47d3a1e-edf7-4b20-b30c-96449721dbef-serving-cert\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342923 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342981 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.342996 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-metrics-certs\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343001 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmds\" (UniqueName: \"kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343038 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt58s\" (UniqueName: \"kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343074 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343105 5778 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered Nov 01 22:23:51.343112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343114 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7jk\" (UniqueName: \"kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343142 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343178 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:51.843158225 +0000 UTC m=+1.619145033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : object "openshift-image-registry"/"installation-pull-secrets" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343196 5778 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343198 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvh6m\" (UniqueName: \"kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m\") pod \"network-check-source-6f78d59f99-xqcfc\" (UID: \"e98acdf9-450d-46a8-b3e8-faae9509a855\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343216 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-stats-auth\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343227 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmz4\" (UniqueName: \"kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343259 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:51.843242412 +0000 UTC m=+1.619229236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : object "openshift-image-registry"/"trusted-ca" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343290 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e707915-61a5-42f8-bb48-920bda324148-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343317 5778 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343332 5778 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fbdb597f4-jpdzl: object "openshift-image-registry"/"image-registry-tls" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.343410 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls podName:d8c3582e-4821-483f-b90a-6175bd9e27be nodeName:}" failed. No retries permitted until 2026-11-01 22:23:51.843394888 +0000 UTC m=+1.619381715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls") pod "image-registry-7fbdb597f4-jpdzl" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be") : object "openshift-image-registry"/"image-registry-tls" not registered Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343456 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-config\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343292 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343513 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1681c152-d4a7-417f-a54e-a7a4b11a2575-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343514 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343583 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.343580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343568 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343619 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343645 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343673 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343701 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343728 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343729 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a0650f02-2be4-4fad-bd28-2fa18f86c6af-default-certificate\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343759 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343785 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p689n\" (UniqueName: \"kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n\") pod \"kas-connection-checker-685895c7c9-5j2lt\" (UID: \"65225fe4-74e2-4b06-9169-7dce163b2fec\") " pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343817 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343841 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343888 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343915 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dngg\" (UniqueName: \"kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343930 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343940 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343965 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.343999 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2hg\" (UniqueName: \"kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344036 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344092 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344120 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344143 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c627c44-b90e-4477-ade0-1a9ce7aafb36-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344146 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344189 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344209 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1681c152-d4a7-417f-a54e-a7a4b11a2575-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2rlrx\" (UID: \"1681c152-d4a7-417f-a54e-a7a4b11a2575\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2rlrx" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344212 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-tmp\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344222 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344272 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344305 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrqx\" (UniqueName: \"kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.344505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344400 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0650f02-2be4-4fad-bd28-2fa18f86c6af-service-ca-bundle\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344547 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-service-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344685 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d47d3a1e-edf7-4b20-b30c-96449721dbef-snapshots\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.344815 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8039013-0ab2-45d6-9321-da3ceb7dc478-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.345303 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c627c44-b90e-4477-ade0-1a9ce7aafb36-config\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.345667 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d47d3a1e-edf7-4b20-b30c-96449721dbef-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.345871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.345718 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc99826a-2d2c-41ae-b55a-e95475056ead-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.346412 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.345923 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc99826a-2d2c-41ae-b55a-e95475056ead-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.346412 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.345968 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aa5e010-5d31-46f0-bb5a-a952613af986-serving-cert\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.353130 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.353110 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5e010-5d31-46f0-bb5a-a952613af986-trusted-ca\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.354524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.354503 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69zf\" (UniqueName: \"kubernetes.io/projected/29c2c4f7-79ec-444c-be78-ce43edf6850c-kube-api-access-l69zf\") pod \"kas-connection-checker-685895c7c9-wtzbw\" (UID: \"29c2c4f7-79ec-444c-be78-ce43edf6850c\") " pod="kube-system/kas-connection-checker-685895c7c9-wtzbw" Nov 01 22:23:51.354645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.354629 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92dfm\" (UniqueName: \"kubernetes.io/projected/0e707915-61a5-42f8-bb48-920bda324148-kube-api-access-92dfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7\" (UID: \"0e707915-61a5-42f8-bb48-920bda324148\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" Nov 01 22:23:51.354721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.354696 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvh6m\" (UniqueName: \"kubernetes.io/projected/e98acdf9-450d-46a8-b3e8-faae9509a855-kube-api-access-tvh6m\") pod \"network-check-source-6f78d59f99-xqcfc\" (UID: \"e98acdf9-450d-46a8-b3e8-faae9509a855\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqcfc" Nov 01 22:23:51.354815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.354798 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmds\" (UniqueName: \"kubernetes.io/projected/d47d3a1e-edf7-4b20-b30c-96449721dbef-kube-api-access-wjmds\") pod \"insights-operator-7c4986d449-nkrwh\" (UID: \"d47d3a1e-edf7-4b20-b30c-96449721dbef\") " pod="openshift-insights/insights-operator-7c4986d449-nkrwh" Nov 01 22:23:51.355074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.355030 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9mz\" (UniqueName: \"kubernetes.io/projected/8aa5e010-5d31-46f0-bb5a-a952613af986-kube-api-access-lg9mz\") pod \"console-operator-7cd56b8dbc-nbf7s\" (UID: \"8aa5e010-5d31-46f0-bb5a-a952613af986\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:51.355433 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.355413 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpx7\" (UniqueName: \"kubernetes.io/projected/06fbbe23-1f70-4c1e-b82f-5fa72a4166db-kube-api-access-mkpx7\") pod \"kas-connection-checker-685895c7c9-jpxjw\" (UID: \"06fbbe23-1f70-4c1e-b82f-5fa72a4166db\") " pod="kube-system/kas-connection-checker-685895c7c9-jpxjw" Nov 01 22:23:51.355877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.355823 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmz4\" (UniqueName: \"kubernetes.io/projected/6c627c44-b90e-4477-ade0-1a9ce7aafb36-kube-api-access-4qmz4\") pod \"service-ca-operator-544d749c8c-p9rls\" (UID: \"6c627c44-b90e-4477-ade0-1a9ce7aafb36\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" Nov 01 22:23:51.355949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.355902 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltb82\" (UniqueName: \"kubernetes.io/projected/3ad608dd-a3a4-47cb-9702-4cbef303f2f7-kube-api-access-ltb82\") pod \"volume-data-source-validator-58c9f5f46d-znttb\" (UID: \"3ad608dd-a3a4-47cb-9702-4cbef303f2f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-znttb" Nov 01 22:23:51.355999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.355931 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7jk\" (UniqueName: \"kubernetes.io/projected/a8039013-0ab2-45d6-9321-da3ceb7dc478-kube-api-access-qj7jk\") pod \"cluster-samples-operator-6ddb6c8b94-msljx\" (UID: \"a8039013-0ab2-45d6-9321-da3ceb7dc478\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-msljx" Nov 01 22:23:51.356352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.356333 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.356979 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.356933 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p689n\" (UniqueName: \"kubernetes.io/projected/65225fe4-74e2-4b06-9169-7dce163b2fec-kube-api-access-p689n\") pod \"kas-connection-checker-685895c7c9-5j2lt\" (UID: \"65225fe4-74e2-4b06-9169-7dce163b2fec\") " pod="kube-system/kas-connection-checker-685895c7c9-5j2lt" Nov 01 22:23:51.357134 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.357114 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2hg\" (UniqueName: \"kubernetes.io/projected/a0650f02-2be4-4fad-bd28-2fa18f86c6af-kube-api-access-gf2hg\") pod \"router-default-7b8b6447f-8bbxr\" (UID: \"a0650f02-2be4-4fad-bd28-2fa18f86c6af\") " pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.357216 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.357141 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrqx\" (UniqueName: \"kubernetes.io/projected/bc99826a-2d2c-41ae-b55a-e95475056ead-kube-api-access-tqrqx\") pod \"cluster-monitoring-operator-884576c57-qsgwz\" (UID: \"bc99826a-2d2c-41ae-b55a-e95475056ead\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qsgwz" Nov 01 22:23:51.426667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.426642 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.426796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.426767 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:23:51.426881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.426804 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.426881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.426813 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.427129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427110 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:23:51.427194 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427109 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:23:51.427194 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427175 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:51.427292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427195 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.427292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427209 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.427292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427221 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:51.427292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427231 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.427292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427277 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:23:51.427496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427366 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.427496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427396 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.427496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427411 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:23:51.427496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427412 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.427676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.427664 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:23:51.428034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428007 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:23:51.428106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428034 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.428106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428073 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.428106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428095 5778 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:23:51.428675 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428652 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:23:51.428743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428684 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.428743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.428688 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.429102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.429086 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 22:23:51.429102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.429093 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:51.429232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.429104 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:23:51.429379 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.429367 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.430078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430060 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:51.430226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430211 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:23:51.430353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430264 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:51.430353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430268 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.430353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430286 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:23:51.430353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430295 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:23:51.430353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430349 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:51.430566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430395 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:23:51.430628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430614 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:23:51.430664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.430611 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:51.431669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.431657 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:23:51.432155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432140 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432163 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432176 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432195 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432178 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d6jqb\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432201 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kgdt\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432220 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432222 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:51.432239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432223 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:23:51.432531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432280 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:23:51.432531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432379 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:23:51.432531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.432398 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:23:51.433164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.433281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433269 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 22:23:51.433357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433344 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:51.433401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433363 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:51.433401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433382 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:23:51.433401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433394 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 22:23:51.433508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433421 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:23:51.433508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433424 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:51.433508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.433424 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.434780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.434761 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3183d19-6697-470b-b5ff-cad0136248fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.435049 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.435032 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt58s\" (UniqueName: \"kubernetes.io/projected/d2372e60-3b4e-407f-b0ed-6799496e9dd7-kube-api-access-lt58s\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.435502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.435481 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.435660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.435642 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.436830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.436809 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dngg\" (UniqueName: \"kubernetes.io/projected/d3183d19-6697-470b-b5ff-cad0136248fa-kube-api-access-4dngg\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.437267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.437239 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.437393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.437380 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 22:23:51.440198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.440182 5778 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:51.440284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.440270 5778 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:51.445315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445299 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445332 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.445392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445350 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445367 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zck\" (UniqueName: \"kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445419 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.445558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445537 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.445653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445578 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.445653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445622 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445651 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-config\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.445729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445675 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-multus-daemon-config\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445690 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.445729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445721 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445739 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445756 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445761 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-tmp\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445773 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445798 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445829 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445887 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.445963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445949 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f84be26-280f-4289-a270-45b31fcd6adc-etc-tuned\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445982 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62bfdd4e-d511-4e4d-a225-17787716d664-konnectivity-ca\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.445994 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446026 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446050 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446076 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446097 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-system-cni-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446113 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446133 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446150 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446170 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9445ebf3-feba-48ea-8eb6-882fd83343ab-original-pull-secret\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446175 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446218 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446241 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.446270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446269 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446359 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446397 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46dx\" (UniqueName: \"kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446432 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62bfdd4e-d511-4e4d-a225-17787716d664-agent-certs\") pod \"konnectivity-agent-w5598\" (UID: \"62bfdd4e-d511-4e4d-a225-17787716d664\") " pod="kube-system/konnectivity-agent-w5598" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446459 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446469 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446530 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446560 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446568 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/010a5ac6-aeef-45ce-8c57-dcebf29427b1-metrics-tls\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446602 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fdk\" (UniqueName: \"kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446637 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446684 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446719 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446769 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446786 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksncs\" (UniqueName: \"kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446802 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446818 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446826 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.446895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446837 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446935 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb90f847-aeda-4739-a6ac-33626b1e17a8-iptables-alerter-script\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.446964 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447024 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447088 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/010a5ac6-aeef-45ce-8c57-dcebf29427b1-config-volume\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447119 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447147 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447170 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447209 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447230 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-ovnkube-script-lib\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447268 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447331 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81b411a3-fb23-4551-894a-958b2404c226-env-overrides\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447345 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5wg\" (UniqueName: \"kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.447390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447392 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447516 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447550 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447576 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5pv\" (UniqueName: \"kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447601 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447636 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447683 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447718 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.447758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447748 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447778 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447808 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447826 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447885 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwbk\" (UniqueName: \"kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447912 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.448013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447956 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.447982 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448069 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448051 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448105 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krx4b\" (UniqueName: \"kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448127 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448144 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448168 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.448227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448164 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63988932-7675-4400-beb1-610e2ecfde16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448279 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448311 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448325 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448342 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448382 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b7p\" (UniqueName: \"kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448418 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.448452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448450 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qtk\" (UniqueName: \"kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448467 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448516 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448606 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb5b9510-e846-4249-911e-6f04e319853f-cni-binary-copy\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448633 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448652 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448670 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.448714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448695 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448724 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448702 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2372e60-3b4e-407f-b0ed-6799496e9dd7-hosts-file\") pod \"node-resolver-t5wh4\" (UID: \"d2372e60-3b4e-407f-b0ed-6799496e9dd7\") " pod="openshift-dns/node-resolver-t5wh4" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448751 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448776 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448809 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk4g\" (UniqueName: \"kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g\") pod \"migrator-66df5d496b-h9fsg\" (UID: \"a6acb24b-4146-452f-b222-008b883c77d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448831 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448869 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448883 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/010a5ac6-aeef-45ce-8c57-dcebf29427b1-tmp-dir\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448896 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448909 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3183d19-6697-470b-b5ff-cad0136248fa-cnibin\") pod \"multus-additional-cni-plugins-h74xz\" (UID: \"d3183d19-6697-470b-b5ff-cad0136248fa\") " pod="openshift-multus/multus-additional-cni-plugins-h74xz" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448926 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448950 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448973 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.448990 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-metrics-certs\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449012 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449049 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.449076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449083 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449082 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449115 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b411a3-fb23-4551-894a-958b2404c226-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449137 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449166 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41572fd2-3250-4c36-a7be-8c8f733c39c3-serviceca\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449192 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63988932-7675-4400-beb1-610e2ecfde16-data-volume\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449253 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49f57d53-5511-4fc4-86b1-a4884a4d213a-cert\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:51.449592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.449284 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63988932-7675-4400-beb1-610e2ecfde16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.452496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.452480 5778 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:51.452550 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.452517 5778 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:51.453974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.453961 5778 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:51.454020 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:23:51.453999 5778 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-223.us-east-2.compute.internal" Nov 01 22:23:51.454319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.454300 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zck\" (UniqueName: \"kubernetes.io/projected/fb5b9510-e846-4249-911e-6f04e319853f-kube-api-access-67zck\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.456309 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.456289 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46dx\" (UniqueName: \"kubernetes.io/projected/010a5ac6-aeef-45ce-8c57-dcebf29427b1-kube-api-access-v46dx\") pod \"dns-default-8zwrr\" (UID: \"010a5ac6-aeef-45ce-8c57-dcebf29427b1\") " pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:51.456352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.456331 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qtk\" (UniqueName: \"kubernetes.io/projected/57c7b714-b4a8-40b1-a4ec-2c0a2117e21c-kube-api-access-p5qtk\") pod \"network-metrics-daemon-jhdh2\" (UID: \"57c7b714-b4a8-40b1-a4ec-2c0a2117e21c\") " pod="openshift-multus/network-metrics-daemon-jhdh2" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457187 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpfb\" (UniqueName: \"kubernetes.io/projected/d5ca1661-6040-419a-9037-1241908c41fa-kube-api-access-4jpfb\") pod \"network-check-target-zvpsn\" (UID: \"d5ca1661-6040-419a-9037-1241908c41fa\") " pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457307 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksncs\" (UniqueName: \"kubernetes.io/projected/63988932-7675-4400-beb1-610e2ecfde16-kube-api-access-ksncs\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457397 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwbk\" (UniqueName: \"kubernetes.io/projected/49f57d53-5511-4fc4-86b1-a4884a4d213a-kube-api-access-7zwbk\") pod \"ingress-canary-wn5v5\" (UID: \"49f57d53-5511-4fc4-86b1-a4884a4d213a\") " pod="openshift-ingress-canary/ingress-canary-wn5v5" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457557 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk4g\" (UniqueName: \"kubernetes.io/projected/a6acb24b-4146-452f-b222-008b883c77d0-kube-api-access-vjk4g\") pod \"migrator-66df5d496b-h9fsg\" (UID: \"a6acb24b-4146-452f-b222-008b883c77d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h9fsg" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457654 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457771 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5pv\" (UniqueName: \"kubernetes.io/projected/bb90f847-aeda-4739-a6ac-33626b1e17a8-kube-api-access-qd5pv\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.457934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457917 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b7p\" (UniqueName: \"kubernetes.io/projected/81b411a3-fb23-4551-894a-958b2404c226-kube-api-access-56b7p\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.458285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.457971 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fdk\" (UniqueName: \"kubernetes.io/projected/5f84be26-280f-4289-a270-45b31fcd6adc-kube-api-access-k6fdk\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.458285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.458118 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"image-registry-5bb5b5746c-v88s9\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:51.458285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.458116 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5wg\" (UniqueName: \"kubernetes.io/projected/41572fd2-3250-4c36-a7be-8c8f733c39c3-kube-api-access-jq5wg\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.458958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.458782 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krx4b\" (UniqueName: \"kubernetes.io/projected/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kube-api-access-krx4b\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550037 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550072 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550088 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550104 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550125 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550167 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41572fd2-3250-4c36-a7be-8c8f733c39c3-host\") pod \"node-ca-xpgg7\" (UID: \"41572fd2-3250-4c36-a7be-8c8f733c39c3\") " pod="openshift-image-registry/node-ca-xpgg7" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550182 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-k8s-cni-cncf-io\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550207 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-ovn\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550232 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550279 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550283 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-socket-dir-parent\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550303 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550286 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550324 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550339 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550359 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550388 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-etc-kubernetes\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550391 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-socket-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550228 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-systemd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550414 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550441 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-systemd\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550447 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-bin\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550448 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-cni-netd\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550458 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550526 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550552 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-lib-modules\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550557 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550561 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysctl-conf\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550582 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550602 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550618 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550628 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-kubelet\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550660 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550631 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550678 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-sysconfig\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550680 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550633 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550707 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-system-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550735 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550775 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-run\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550780 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550809 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550827 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550844 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550881 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550922 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-var-lib-kubelet\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.550907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550927 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550970 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-sys\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550971 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63988932-7675-4400-beb1-610e2ecfde16-crio-socket\") pod \"insights-runtime-extractor-2jgr7\" (UID: \"63988932-7675-4400-beb1-610e2ecfde16\") " pod="openshift-insights/insights-runtime-extractor-2jgr7" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550985 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-host\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550939 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550943 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-multus-certs\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.550997 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551051 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551075 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551101 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551054 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-cni-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551106 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-cnibin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551130 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-device-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551154 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-var-lib-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551155 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551166 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-log-socket\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551187 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-run-netns\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551196 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551224 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-kubernetes\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551247 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551272 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551295 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551322 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551339 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-hostroot\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551350 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551375 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-node-log\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551374 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-sys-fs\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551376 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551408 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-kubelet\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551381 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb90f847-aeda-4739-a6ac-33626b1e17a8-host-slash\") pod \"iptables-alerter-54bgs\" (UID: \"bb90f847-aeda-4739-a6ac-33626b1e17a8\") " pod="openshift-network-operator/iptables-alerter-54bgs" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551424 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551457 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551472 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-dbus\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551478 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551507 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551543 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551566 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f84be26-280f-4289-a270-45b31fcd6adc-etc-modprobe-d\") pod \"tuned-ghlmj\" (UID: \"5f84be26-280f-4289-a270-45b31fcd6adc\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlmj" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551567 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551621 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551632 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-host-slash\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551646 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551669 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-bin\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551598 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-run-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551715 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-var-lib-cni-multus\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551714 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551379 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-multus-conf-dir\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551745 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb5b9510-e846-4249-911e-6f04e319853f-host-run-netns\") pod \"multus-7jrrm\" (UID: \"fb5b9510-e846-4249-911e-6f04e319853f\") " pod="openshift-multus/multus-7jrrm" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551759 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-systemd-units\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551670 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9445ebf3-feba-48ea-8eb6-882fd83343ab-kubelet-config\") pod \"global-pull-secret-syncer-vhgqk\" (UID: \"9445ebf3-feba-48ea-8eb6-882fd83343ab\") " pod="kube-system/global-pull-secret-syncer-vhgqk" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551788 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551790 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81b411a3-fb23-4551-894a-958b2404c226-etc-openvswitch\") pod \"ovnkube-node-pj8xg\" (UID: \"81b411a3-fb23-4551-894a-958b2404c226\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:51.552247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.551888 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c545d6bc-16aa-4c0d-b30e-2ac42f675347-registration-dir\") pod \"aws-ebs-csi-driver-node-wjqc4\" (UID: \"c545d6bc-16aa-4c0d-b30e-2ac42f675347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjqc4" Nov 01 22:23:51.625402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.625330 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:51.647011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.646985 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:23:51.853594 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.853550 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.853758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.853612 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.853758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.853656 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.853936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.853915 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.853978 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.853938 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:51.854101 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:51.854084 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"image-registry-7fbdb597f4-jpdzl\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:23:52.445554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:52.445524 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jgr7" event={"ID":"63988932-7675-4400-beb1-610e2ecfde16","Type":"ContainerStarted","Data":"6d76dc9002a3840ade8e5d03f339071f9ccf182805066225f69b8e03bb3c33f1"} Nov 01 22:23:52.627124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:52.627045 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:53.307677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:53.307640 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w5598" Nov 01 22:23:53.451481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:53.451387 5778 generic.go:356] "Generic (PLEG): container finished" podID="6c627c44-b90e-4477-ade0-1a9ce7aafb36" containerID="7c56edabb3eefb4b2292e135c9bdec28eaedcdcfae0451b5cfecee3bd3741f2a" exitCode=0 Nov 01 22:23:53.451481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:53.451411 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" event={"ID":"6c627c44-b90e-4477-ade0-1a9ce7aafb36","Type":"ContainerDied","Data":"7c56edabb3eefb4b2292e135c9bdec28eaedcdcfae0451b5cfecee3bd3741f2a"} Nov 01 22:23:53.451942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:53.451926 5778 scope.go:122] "RemoveContainer" containerID="7c56edabb3eefb4b2292e135c9bdec28eaedcdcfae0451b5cfecee3bd3741f2a" Nov 01 22:23:53.644953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:53.644921 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:23:54.083510 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.083483 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:54.457580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.457546 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jgr7" event={"ID":"63988932-7675-4400-beb1-610e2ecfde16","Type":"ContainerStarted","Data":"41ccb5bece4abb1964e9013062e3eb13967b670571de6c9fea4d7663c3adbcbd"} Nov 01 22:23:54.459584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.459558 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rls" event={"ID":"6c627c44-b90e-4477-ade0-1a9ce7aafb36","Type":"ContainerStarted","Data":"9e93c7ca107afd4d0907fa183ae1f4034e7ba596029462bcf5a6ead9d9c72788"} Nov 01 22:23:54.461071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.461042 5778 generic.go:356] "Generic (PLEG): container finished" podID="0e707915-61a5-42f8-bb48-920bda324148" containerID="854f193bce62849627de13556755f7c3f3ce3962dea2077cc720152cbc94ce8d" exitCode=0 Nov 01 22:23:54.461189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.461068 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" event={"ID":"0e707915-61a5-42f8-bb48-920bda324148","Type":"ContainerDied","Data":"854f193bce62849627de13556755f7c3f3ce3962dea2077cc720152cbc94ce8d"} Nov 01 22:23:54.461360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:54.461348 5778 scope.go:122] "RemoveContainer" containerID="854f193bce62849627de13556755f7c3f3ce3962dea2077cc720152cbc94ce8d" Nov 01 22:23:55.215319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.215281 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:55.215548 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.215419 5778 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:23:55.216643 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.216619 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b8b6447f-8bbxr" Nov 01 22:23:55.469801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.469666 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5bf7" event={"ID":"0e707915-61a5-42f8-bb48-920bda324148","Type":"ContainerStarted","Data":"b25833f6a42846cab8b969d660159000ba6338738fd08724ed2fb9cd706810b3"} Nov 01 22:23:55.682964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.682932 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r4d2n"] Nov 01 22:23:55.687000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.686978 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.694920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.694894 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:23:55.694920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.694919 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:23:55.695123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.694960 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:23:55.695123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.694974 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:55.695123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.695044 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5n9np\"" Nov 01 22:23:55.791148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791064 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2md\" (UniqueName: \"kubernetes.io/projected/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-kube-api-access-kb2md\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791101 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-root\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791128 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791195 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-textfile\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791283 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-wtmp\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791310 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-tls\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791387 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791417 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-metrics-client-ca\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.791492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.791435 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-sys\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.892823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892790 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-wtmp\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.892823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892834 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-tls\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892893 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892926 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-metrics-client-ca\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892951 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-sys\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.892998 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2md\" (UniqueName: \"kubernetes.io/projected/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-kube-api-access-kb2md\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893008 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-wtmp\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893023 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-root\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893065 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-root\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893064 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-sys\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893080 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893512 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893193 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-textfile\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893512 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893447 5778 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:23:55.893616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893505 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-textfile\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893587 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-metrics-client-ca\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.893731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.893691 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.895652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.895625 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-tls\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.895758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.895671 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.905733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.905710 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2md\" (UniqueName: \"kubernetes.io/projected/2776ec9b-f80e-438f-a30b-53e86e5b4f2f-kube-api-access-kb2md\") pod \"node-exporter-r4d2n\" (UID: \"2776ec9b-f80e-438f-a30b-53e86e5b4f2f\") " pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:55.940034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:55.940003 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nbf7s" Nov 01 22:23:56.003191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.003159 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4d2n" Nov 01 22:23:56.016030 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:23:56.015981 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2776ec9b_f80e_438f_a30b_53e86e5b4f2f.slice/crio-42ece44bcc17ce29ae5efc2186b38e13b081aa21654dfb1a29320c2cf87cb887 WatchSource:0}: Error finding container 42ece44bcc17ce29ae5efc2186b38e13b081aa21654dfb1a29320c2cf87cb887: Status 404 returned error can't find the container with id 42ece44bcc17ce29ae5efc2186b38e13b081aa21654dfb1a29320c2cf87cb887 Nov 01 22:23:56.199792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.199766 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:23:56.475043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.474955 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4d2n" event={"ID":"2776ec9b-f80e-438f-a30b-53e86e5b4f2f","Type":"ContainerStarted","Data":"42ece44bcc17ce29ae5efc2186b38e13b081aa21654dfb1a29320c2cf87cb887"} Nov 01 22:23:56.476445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.476421 5778 generic.go:356] "Generic (PLEG): container finished" podID="d47d3a1e-edf7-4b20-b30c-96449721dbef" containerID="59a0189d8e164b160e0c396d27afc7ae0f98afdfa4a3189f06453c489fc65199" exitCode=0 Nov 01 22:23:56.476570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.476475 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" event={"ID":"d47d3a1e-edf7-4b20-b30c-96449721dbef","Type":"ContainerDied","Data":"59a0189d8e164b160e0c396d27afc7ae0f98afdfa4a3189f06453c489fc65199"} Nov 01 22:23:56.476793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:56.476773 5778 scope.go:122] "RemoveContainer" containerID="59a0189d8e164b160e0c396d27afc7ae0f98afdfa4a3189f06453c489fc65199" Nov 01 22:23:57.086918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:57.086882 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8zwrr" Nov 01 22:23:57.483034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:57.482995 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4d2n" event={"ID":"2776ec9b-f80e-438f-a30b-53e86e5b4f2f","Type":"ContainerStarted","Data":"91ae295ba2f625a304d5f80879d7fd2cce9735ad1c8a95e48adbc013c12a6538"} Nov 01 22:23:57.485636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:57.485611 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nkrwh" event={"ID":"d47d3a1e-edf7-4b20-b30c-96449721dbef","Type":"ContainerStarted","Data":"a3da690f6b2af4ea0e1ef30704a4db1d0d388b65a78903738370727ea838a84e"} Nov 01 22:23:57.598448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:57.598417 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:23:58.491307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:58.491274 5778 generic.go:356] "Generic (PLEG): container finished" podID="2776ec9b-f80e-438f-a30b-53e86e5b4f2f" containerID="91ae295ba2f625a304d5f80879d7fd2cce9735ad1c8a95e48adbc013c12a6538" exitCode=0 Nov 01 22:23:58.491714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:58.491356 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4d2n" event={"ID":"2776ec9b-f80e-438f-a30b-53e86e5b4f2f","Type":"ContainerDied","Data":"91ae295ba2f625a304d5f80879d7fd2cce9735ad1c8a95e48adbc013c12a6538"} Nov 01 22:23:59.417601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.417572 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2rlrx_1681c152-d4a7-417f-a54e-a7a4b11a2575/networking-console-plugin/0.log" Nov 01 22:23:59.497472 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.497431 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4d2n" event={"ID":"2776ec9b-f80e-438f-a30b-53e86e5b4f2f","Type":"ContainerStarted","Data":"16fd6ce3c4f74cb5695c4bbf077713e800886c93d540f407b527123d9effbfcf"} Nov 01 22:23:59.497900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.497501 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4d2n" event={"ID":"2776ec9b-f80e-438f-a30b-53e86e5b4f2f","Type":"ContainerStarted","Data":"6a2d66ba973f4f5afda691f3b90eedc4e036699b3ac0b9182dbbef22b05cf2d3"} Nov 01 22:23:59.518362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.518313 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r4d2n" podStartSLOduration=3.092354483 podStartE2EDuration="4.51829921s" totalImagesPullingTime="1.349337484s" totalInitContainerRuntime="76.607243ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:59.516443361 +0000 UTC m=+9.292430209" watchObservedRunningTime="2026-11-01 22:23:59.51829921 +0000 UTC m=+9.294286073" Nov 01 22:23:59.601260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.601235 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:23:59.803718 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:23:59.803637 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/2.log" Nov 01 22:24:00.409562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:00.409521 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:24:00.409786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:00.409770 5778 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:24:01.000470 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:01.000438 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zwrr_010a5ac6-aeef-45ce-8c57-dcebf29427b1/dns/0.log" Nov 01 22:24:01.200965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:01.200927 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zwrr_010a5ac6-aeef-45ce-8c57-dcebf29427b1/kube-rbac-proxy/0.log" Nov 01 22:24:01.508801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:01.508774 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvpsn" Nov 01 22:24:02.001266 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:02.001236 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5wh4_d2372e60-3b4e-407f-b0ed-6799496e9dd7/dns-node-resolver/0.log" Nov 01 22:24:02.401983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:02.401901 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b8b6447f-8bbxr_a0650f02-2be4-4fad-bd28-2fa18f86c6af/router/0.log" Nov 01 22:24:03.004068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:03.004033 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wn5v5_49f57d53-5511-4fc4-86b1-a4884a4d213a/serve-healthcheck-canary/0.log" Nov 01 22:24:16.350803 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.350721 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" podUID="d8c3582e-4821-483f-b90a-6175bd9e27be" containerName="registry" containerID="cri-o://449549da8aa4e9704cf2b2618796939b2cf090c3e1221648c916ab2849199701" gracePeriod=30 Nov 01 22:24:16.561912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.561880 5778 generic.go:356] "Generic (PLEG): container finished" podID="d8c3582e-4821-483f-b90a-6175bd9e27be" containerID="449549da8aa4e9704cf2b2618796939b2cf090c3e1221648c916ab2849199701" exitCode=0 Nov 01 22:24:16.562083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.561955 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" event={"ID":"d8c3582e-4821-483f-b90a-6175bd9e27be","Type":"ContainerDied","Data":"449549da8aa4e9704cf2b2618796939b2cf090c3e1221648c916ab2849199701"} Nov 01 22:24:16.629424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.629352 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:24:16.779286 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779254 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779326 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779368 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779395 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779560 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779627 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779666 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779717 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") pod \"d8c3582e-4821-483f-b90a-6175bd9e27be\" (UID: \"d8c3582e-4821-483f-b90a-6175bd9e27be\") " Nov 01 22:24:16.779737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779721 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:16.780222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.779793 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:16.780748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.780358 5778 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-trusted-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.780748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.780386 5778 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-certificates\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.782291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.782247 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:16.782393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.782292 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:16.782509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.782468 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "kube-api-access-sstm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:16.782612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.782550 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:16.782612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.782589 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:16.787126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.787094 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted" pod "d8c3582e-4821-483f-b90a-6175bd9e27be" (UID: "d8c3582e-4821-483f-b90a-6175bd9e27be"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880741 5778 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-bound-sa-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880768 5778 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-image-registry-private-configuration\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880779 5778 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-registry-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880788 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstm7\" (UniqueName: \"kubernetes.io/projected/d8c3582e-4821-483f-b90a-6175bd9e27be-kube-api-access-sstm7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880797 5778 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8c3582e-4821-483f-b90a-6175bd9e27be-ca-trust-extracted\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:16.880815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:16.880806 5778 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8c3582e-4821-483f-b90a-6175bd9e27be-installation-pull-secrets\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:17.567112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.567082 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" Nov 01 22:24:17.567112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.567090 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbdb597f4-jpdzl" event={"ID":"d8c3582e-4821-483f-b90a-6175bd9e27be","Type":"ContainerDied","Data":"39ff92eb6f4697322575185692ff531fcef940f0a942a8f3ce8bd09a7ec6fcfd"} Nov 01 22:24:17.567621 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.567142 5778 scope.go:122] "RemoveContainer" containerID="449549da8aa4e9704cf2b2618796939b2cf090c3e1221648c916ab2849199701" Nov 01 22:24:17.591981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.591951 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:24:17.599385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.599358 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fbdb597f4-jpdzl"] Nov 01 22:24:17.878801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:17.878727 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8xg" Nov 01 22:24:18.374663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:18.372671 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c3582e-4821-483f-b90a-6175bd9e27be" path="/var/lib/kubelet/pods/d8c3582e-4821-483f-b90a-6175bd9e27be/volumes" Nov 01 22:24:18.772086 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:24:18.772042 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:24:21.220099 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.220048 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" containerID="cri-o://c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897" gracePeriod=30 Nov 01 22:24:21.465029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.465004 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:24:21.529932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.529835 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.529932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.529887 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.529932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.529921 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.529959 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.529993 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.530024 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.530046 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.530086 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") pod \"24621246-44ab-4de5-a8a8-389f98b09d22\" (UID: \"24621246-44ab-4de5-a8a8-389f98b09d22\") " Nov 01 22:24:21.530444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.530414 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:21.530545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.530519 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:21.532658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.532631 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "kube-api-access-wnc6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:21.532934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.532902 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:21.533051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.533013 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:21.533223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.533181 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:21.533223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.533191 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:21.536769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.536745 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted" pod "24621246-44ab-4de5-a8a8-389f98b09d22" (UID: "24621246-44ab-4de5-a8a8-389f98b09d22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:21.586223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.586195 5778 generic.go:356] "Generic (PLEG): container finished" podID="24621246-44ab-4de5-a8a8-389f98b09d22" containerID="c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897" exitCode=0 Nov 01 22:24:21.586333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.586243 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" Nov 01 22:24:21.586333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.586262 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" event={"ID":"24621246-44ab-4de5-a8a8-389f98b09d22","Type":"ContainerDied","Data":"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897"} Nov 01 22:24:21.586333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.586301 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5b5746c-v88s9" event={"ID":"24621246-44ab-4de5-a8a8-389f98b09d22","Type":"ContainerDied","Data":"038f1f74ca4041ef70194943f33d90afce4063fb5bf9cd29da50f3d43261f1ed"} Nov 01 22:24:21.586333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.586321 5778 scope.go:122] "RemoveContainer" containerID="c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897" Nov 01 22:24:21.595127 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.595110 5778 scope.go:122] "RemoveContainer" containerID="c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897" Nov 01 22:24:21.595374 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:24:21.595354 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897\": container with ID starting with c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897 not found: ID does not exist" containerID="c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897" Nov 01 22:24:21.595422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.595388 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897"} err="failed to get container status \"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897\": rpc error: code = NotFound desc = could not find container \"c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897\": container with ID starting with c5ef920bca7e875cb533986e9d9c017f9b33c1613449ca31f506078b40e75897 not found: ID does not exist" Nov 01 22:24:21.606574 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.606552 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:24:21.616692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.616673 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bb5b5746c-v88s9"] Nov 01 22:24:21.630548 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630524 5778 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24621246-44ab-4de5-a8a8-389f98b09d22-ca-trust-extracted\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630549 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc6h\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-kube-api-access-wnc6h\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630566 5778 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-registry-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630574 5778 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-registry-certificates\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630584 5778 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-image-registry-private-configuration\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630592 5778 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24621246-44ab-4de5-a8a8-389f98b09d22-installation-pull-secrets\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630601 5778 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24621246-44ab-4de5-a8a8-389f98b09d22-trusted-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:21.630630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:21.630610 5778 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24621246-44ab-4de5-a8a8-389f98b09d22-bound-sa-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:22.371979 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:24:22.371939 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" path="/var/lib/kubelet/pods/24621246-44ab-4de5-a8a8-389f98b09d22/volumes" Nov 01 22:24:48.771369 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:24:48.771330 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:25:18.771861 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:25:18.771796 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:25:48.771766 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:25:48.771725 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:26:18.771387 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:26:18.771344 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:26:48.772369 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:26:48.772319 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:27:18.772018 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:27:18.771979 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:27:48.772754 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:27:48.772715 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:28:18.772096 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:28:18.772056 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:28:48.772395 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:28:48.772356 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:28:50.386747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:28:50.386720 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:28:50.387231 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:28:50.387073 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:29:13.120549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120517 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn"] Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120821 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d8c3582e-4821-483f-b90a-6175bd9e27be" containerName="registry" Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120833 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c3582e-4821-483f-b90a-6175bd9e27be" containerName="registry" Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120843 5778 cpu_manager.go:414] "containerMap: removing container" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120863 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120921 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c3582e-4821-483f-b90a-6175bd9e27be" containerName="registry" Nov 01 22:29:13.121044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.120931 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="24621246-44ab-4de5-a8a8-389f98b09d22" containerName="registry" Nov 01 22:29:13.124083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.124067 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.136646 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.136618 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn"] Nov 01 22:29:13.139169 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.139145 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4ltq4\"" Nov 01 22:29:13.139478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.139463 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 22:29:13.139633 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.139618 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 22:29:13.176014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.175985 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.176170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.176029 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgtm\" (UniqueName: \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.176170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.176055 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.277054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.277019 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.277248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.277071 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgtm\" (UniqueName: \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.277248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.277165 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.277484 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.277465 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.277554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.277497 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.286053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.286034 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgtm\" (UniqueName: \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.439636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.439609 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:13.716246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.716175 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn"] Nov 01 22:29:13.720224 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:13.720196 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2e05e8_a7db_4cb4_99a5_21dcb69d077a.slice/crio-64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9 WatchSource:0}: Error finding container 64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9: Status 404 returned error can't find the container with id 64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9 Nov 01 22:29:13.722545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.722528 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:29:13.733761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.733729 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp"] Nov 01 22:29:13.740767 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.740745 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.744981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.744956 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp"] Nov 01 22:29:13.745436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.745417 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2r9h4\"" Nov 01 22:29:13.781458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.781427 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6xw\" (UniqueName: \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.781575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.781473 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.781618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.781579 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.833341 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.833314 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb"] Nov 01 22:29:13.836589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.836574 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.843628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.843608 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ch974\"" Nov 01 22:29:13.847646 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.847623 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb"] Nov 01 22:29:13.881968 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.881940 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6xw\" (UniqueName: \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.882096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.881982 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.882096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882007 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.882096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882033 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xfj\" (UniqueName: \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.882096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882060 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.882096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882094 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.882397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882326 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.882491 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.882439 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.893764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.893743 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6xw\" (UniqueName: \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:13.943700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.943669 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn"] Nov 01 22:29:13.947227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.947210 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:13.950727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.950705 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-nbzh7\"" Nov 01 22:29:13.955663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.955643 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn"] Nov 01 22:29:13.983509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983456 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.983509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983486 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xfj\" (UniqueName: \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.983658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983517 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.983658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983552 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wg8\" (UniqueName: \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:13.983658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983575 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:13.983658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983592 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:13.983900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983875 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.983941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.983898 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:13.993376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:13.993355 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xfj\" (UniqueName: \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:14.043182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.043161 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg"] Nov 01 22:29:14.046728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.046714 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.050414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.050393 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-sl2cc\"" Nov 01 22:29:14.055151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.055127 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg"] Nov 01 22:29:14.057942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.057923 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:14.084181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084156 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.084302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084192 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wg8\" (UniqueName: \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.084302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084216 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.084302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084238 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.084463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084301 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.084506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084458 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjh68\" (UniqueName: \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.084622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084603 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.084756 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.084729 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.093564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.093543 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wg8\" (UniqueName: \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.152453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.152420 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:14.186238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.185764 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.186238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.185818 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjh68\" (UniqueName: \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.186238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.185892 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.186238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.186216 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.186238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.186218 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.195763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.195731 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjh68\" (UniqueName: \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.264109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.264025 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:14.329843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.329817 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp"] Nov 01 22:29:14.363654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.363616 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:14.431081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.431053 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb"] Nov 01 22:29:14.436334 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:14.436303 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2e1710_8428_4634_bf53_89efdbfbe2ed.slice/crio-bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef WatchSource:0}: Error finding container bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef: Status 404 returned error can't find the container with id bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef Nov 01 22:29:14.553346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.553266 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn"] Nov 01 22:29:14.557129 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:14.557095 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fd497b_6f87_4b1b_bea8_7ce004ddf8a9.slice/crio-205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951 WatchSource:0}: Error finding container 205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951: Status 404 returned error can't find the container with id 205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951 Nov 01 22:29:14.612427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.612366 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" event={"ID":"08b772bd-d151-42b6-a2f5-7b93e1434970","Type":"ContainerStarted","Data":"777b82175ba1dff58b18d4311c9ae1f5a0f7e95fdf1ef87fa1a0a52764030faa"} Nov 01 22:29:14.613731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.613700 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" event={"ID":"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9","Type":"ContainerStarted","Data":"205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951"} Nov 01 22:29:14.615030 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.614997 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" event={"ID":"7a2e1710-8428-4634-bf53-89efdbfbe2ed","Type":"ContainerStarted","Data":"bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef"} Nov 01 22:29:14.616312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.616287 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" event={"ID":"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a","Type":"ContainerStarted","Data":"64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9"} Nov 01 22:29:14.659177 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:14.659150 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg"] Nov 01 22:29:14.661753 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:14.661708 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae16d4a_b3e1_4b50_988d_69270b4584da.slice/crio-f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2 WatchSource:0}: Error finding container f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2: Status 404 returned error can't find the container with id f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2 Nov 01 22:29:15.624728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.624630 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerStarted","Data":"f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2"} Nov 01 22:29:15.709525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.709492 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 22:29:15.713367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.713338 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.726246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.725811 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 22:29:15.726246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.725949 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27s9n\"" Nov 01 22:29:15.726749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.726363 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 22:29:15.803645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803612 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.803645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803651 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803681 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803730 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803758 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803788 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803864 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803915 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.809070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.803945 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.815924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.815887 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp"] Nov 01 22:29:15.819570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.819550 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.824731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.824708 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-lbcms\"" Nov 01 22:29:15.845871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.845734 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp"] Nov 01 22:29:15.905378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905338 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w574v\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-kube-api-access-w574v\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905403 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905439 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da3106a-9735-4255-8a62-a2696577b969-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905466 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905494 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905538 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905562 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905601 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da3106a-9735-4255-8a62-a2696577b969-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905637 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905665 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905702 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905731 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905756 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905808 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905835 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905877 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905904 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:15.905973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.905936 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.906486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.906372 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.907411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.907387 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.907658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.907639 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.908314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.908288 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.908553 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.908533 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.913461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.913434 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.917299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.917273 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.918972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.918924 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:15.920074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:15.920051 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7k2sf\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006586 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da3106a-9735-4255-8a62-a2696577b969-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006657 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006693 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da3106a-9735-4255-8a62-a2696577b969-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006748 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006773 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006799 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006844 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006885 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.006923 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w574v\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-kube-api-access-w574v\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.008023 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.008758 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da3106a-9735-4255-8a62-a2696577b969-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.009004 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.009023 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.009760 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.009253 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.019057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.019025 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da3106a-9735-4255-8a62-a2696577b969-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.019989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.019962 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da3106a-9735-4255-8a62-a2696577b969-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.027433 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.027405 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.027778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.027720 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w574v\" (UniqueName: \"kubernetes.io/projected/6da3106a-9735-4255-8a62-a2696577b969-kube-api-access-w574v\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp\" (UID: \"6da3106a-9735-4255-8a62-a2696577b969\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.034634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.034610 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:16.151331 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.151215 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:16.341474 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.339602 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 22:29:16.343982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:16.343950 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff8306b_24d3_4d57_b45f_bad4ce1f906b.slice/crio-12feb6c0f2c5cae67cc5bcea79c4887fffc1afb217be7a6959b899e40f3aaafb WatchSource:0}: Error finding container 12feb6c0f2c5cae67cc5bcea79c4887fffc1afb217be7a6959b899e40f3aaafb: Status 404 returned error can't find the container with id 12feb6c0f2c5cae67cc5bcea79c4887fffc1afb217be7a6959b899e40f3aaafb Nov 01 22:29:16.471569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.471490 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp"] Nov 01 22:29:16.481253 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:16.481221 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da3106a_9735_4255_8a62_a2696577b969.slice/crio-223b49a7b2d84139b1bbef6b68d4a5a0894b791306db24d314e261a7b05f1dab WatchSource:0}: Error finding container 223b49a7b2d84139b1bbef6b68d4a5a0894b791306db24d314e261a7b05f1dab: Status 404 returned error can't find the container with id 223b49a7b2d84139b1bbef6b68d4a5a0894b791306db24d314e261a7b05f1dab Nov 01 22:29:16.485200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.485097 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt"] Nov 01 22:29:16.492240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.491433 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.506835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.506811 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-b7w4j\"" Nov 01 22:29:16.524787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.524753 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt"] Nov 01 22:29:16.619141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.619071 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.619396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.619177 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.619396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.619217 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5jw\" (UniqueName: \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.633108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.633072 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" event={"ID":"6da3106a-9735-4255-8a62-a2696577b969","Type":"ContainerStarted","Data":"223b49a7b2d84139b1bbef6b68d4a5a0894b791306db24d314e261a7b05f1dab"} Nov 01 22:29:16.635409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.635341 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerStarted","Data":"12feb6c0f2c5cae67cc5bcea79c4887fffc1afb217be7a6959b899e40f3aaafb"} Nov 01 22:29:16.720489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.720449 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.720663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.720547 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.720663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.720581 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5jw\" (UniqueName: \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.721300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.721267 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.721485 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.721470 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.732791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.732717 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5jw\" (UniqueName: \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:16.814385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:16.814123 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:17.144007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.143930 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk"] Nov 01 22:29:17.149761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.149733 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt"] Nov 01 22:29:17.149967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.149871 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.158556 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:17.156264 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe9c226_fa93_4fc8_b0bd_95a33379b743.slice/crio-0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5 WatchSource:0}: Error finding container 0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5: Status 404 returned error can't find the container with id 0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5 Nov 01 22:29:17.159394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.158912 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk"] Nov 01 22:29:17.160571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.160539 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7vtnw\"" Nov 01 22:29:17.225468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.225435 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.225663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.225492 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvtg\" (UniqueName: \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.225663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.225611 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.326893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.326841 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.327093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.326930 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.327093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.326993 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvtg\" (UniqueName: \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.328150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.328095 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.328585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.328547 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.345966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.345938 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvtg\" (UniqueName: \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.477593 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.477262 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:17.661985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.661941 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" event={"ID":"efe9c226-fa93-4fc8-b0bd-95a33379b743","Type":"ContainerStarted","Data":"0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5"} Nov 01 22:29:17.780215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:17.780060 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk"] Nov 01 22:29:18.677787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:18.677747 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" event={"ID":"f176af06-3cf2-44bd-85fa-e1aabe357631","Type":"ContainerStarted","Data":"e7a6699edaa1b8dfe84a63031a7320957dc2b93553655ac32d60752f85d358a0"} Nov 01 22:29:18.809101 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:29:18.809019 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:29:24.347421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:24.347370 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:24.347815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:24.347436 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:24.347815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:24.347483 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:24.746797 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:24.746752 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerStarted","Data":"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24"} Nov 01 22:29:25.020037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.019472 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.020037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.019570 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.020037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.019613 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.036030 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.035922 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:25.041741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.041716 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:25.060272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.060216 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podStartSLOduration=2.063227718 podStartE2EDuration="10.060199448s" totalImagesPullingTime="7.99697173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:24.769019956 +0000 UTC m=+334.545006786" watchObservedRunningTime="2026-11-01 22:29:25.060199448 +0000 UTC m=+334.836186275" Nov 01 22:29:25.758064 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.758025 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" event={"ID":"6da3106a-9735-4255-8a62-a2696577b969","Type":"ContainerStarted","Data":"7d736adab3da6c80a4d86efe86f2e04eb6b9fc29786acb709707dedfce8d0363"} Nov 01 22:29:25.758538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.758128 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:25.759810 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.759786 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:29:25.787678 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:25.786189 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" podStartSLOduration=2.2557040329999998 podStartE2EDuration="10.786169939s" totalImagesPullingTime="8.530465906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:25.783383122 +0000 UTC m=+335.559369952" watchObservedRunningTime="2026-11-01 22:29:25.786169939 +0000 UTC m=+335.562156773" Nov 01 22:29:26.154298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:26.152954 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:26.159797 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:26.159588 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:26.765529 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:26.765487 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:26.766882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:26.766651 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp" Nov 01 22:29:28.780360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:28.779622 5778 generic.go:356] "Generic (PLEG): container finished" podID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerID="c312a2077b559cde8c08f409edb896d09adb6622ed21ebd5f5eb2fec649b16c0" exitCode=0 Nov 01 22:29:28.780360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:28.779913 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" event={"ID":"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a","Type":"ContainerDied","Data":"c312a2077b559cde8c08f409edb896d09adb6622ed21ebd5f5eb2fec649b16c0"} Nov 01 22:29:29.785480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.785443 5778 generic.go:356] "Generic (PLEG): container finished" podID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerID="688f99d10281329e8808aa564d9f3a26de9a330e5deae78eb64ba8ca7e707a70" exitCode=0 Nov 01 22:29:29.785942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.785535 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerDied","Data":"688f99d10281329e8808aa564d9f3a26de9a330e5deae78eb64ba8ca7e707a70"} Nov 01 22:29:29.787216 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.787189 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerID="bca963572802e3007799123730ac06fb2741b01d22d2b6fe19974f562c8ac504" exitCode=0 Nov 01 22:29:29.787347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.787295 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" event={"ID":"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9","Type":"ContainerDied","Data":"bca963572802e3007799123730ac06fb2741b01d22d2b6fe19974f562c8ac504"} Nov 01 22:29:29.789158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.789082 5778 generic.go:356] "Generic (PLEG): container finished" podID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerID="1321193a4da497ad70858045cd3d2c4e411a2b46f1cdeb52ac2d80de3d5b1ca1" exitCode=0 Nov 01 22:29:29.789239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.789214 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" event={"ID":"7a2e1710-8428-4634-bf53-89efdbfbe2ed","Type":"ContainerDied","Data":"1321193a4da497ad70858045cd3d2c4e411a2b46f1cdeb52ac2d80de3d5b1ca1"} Nov 01 22:29:29.790821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.790799 5778 generic.go:356] "Generic (PLEG): container finished" podID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerID="bee76e7a3fb373882613c161422b688d2279bc04ee18077cd94f692ad15775a7" exitCode=0 Nov 01 22:29:29.790924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.790887 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" event={"ID":"08b772bd-d151-42b6-a2f5-7b93e1434970","Type":"ContainerDied","Data":"bee76e7a3fb373882613c161422b688d2279bc04ee18077cd94f692ad15775a7"} Nov 01 22:29:29.792471 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.792449 5778 generic.go:356] "Generic (PLEG): container finished" podID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerID="cfad17ad723f9af4dd4662c2607f6e8a4eb0a348e3cfc814c01c3802542d5d47" exitCode=0 Nov 01 22:29:29.792554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:29.792490 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" event={"ID":"efe9c226-fa93-4fc8-b0bd-95a33379b743","Type":"ContainerDied","Data":"cfad17ad723f9af4dd4662c2607f6e8a4eb0a348e3cfc814c01c3802542d5d47"} Nov 01 22:29:30.799065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.799030 5778 generic.go:356] "Generic (PLEG): container finished" podID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerID="70acdfe631acbe721900036b63b6e51b9c1a0abe05976bb4f8b3662af1bc896c" exitCode=0 Nov 01 22:29:30.799489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.799119 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" event={"ID":"08b772bd-d151-42b6-a2f5-7b93e1434970","Type":"ContainerDied","Data":"70acdfe631acbe721900036b63b6e51b9c1a0abe05976bb4f8b3662af1bc896c"} Nov 01 22:29:30.801490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.801451 5778 generic.go:356] "Generic (PLEG): container finished" podID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerID="fa3512e19103fc36b21c912c9f5d446d557e5d25fb4d97b6dd647749282d9f08" exitCode=0 Nov 01 22:29:30.801612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.801505 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" event={"ID":"efe9c226-fa93-4fc8-b0bd-95a33379b743","Type":"ContainerDied","Data":"fa3512e19103fc36b21c912c9f5d446d557e5d25fb4d97b6dd647749282d9f08"} Nov 01 22:29:30.803251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.803228 5778 generic.go:356] "Generic (PLEG): container finished" podID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerID="25076866aa997a1b81b5d8689514913fae127deaa5986087147caf91e05c6f61" exitCode=0 Nov 01 22:29:30.803341 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:30.803278 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" event={"ID":"f176af06-3cf2-44bd-85fa-e1aabe357631","Type":"ContainerDied","Data":"25076866aa997a1b81b5d8689514913fae127deaa5986087147caf91e05c6f61"} Nov 01 22:29:31.814280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.814242 5778 generic.go:356] "Generic (PLEG): container finished" podID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerID="18eb3650072e8766d30e01239b9d25172a54727bb4757fff1d5ae186af415577" exitCode=0 Nov 01 22:29:31.814760 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.814335 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerDied","Data":"18eb3650072e8766d30e01239b9d25172a54727bb4757fff1d5ae186af415577"} Nov 01 22:29:31.819714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.819683 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerID="dc564db8298b19c27537fbef1c6bf5765bca7f23ed911c007e6ff3285ea1b8f0" exitCode=0 Nov 01 22:29:31.819833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.819798 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" event={"ID":"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9","Type":"ContainerDied","Data":"dc564db8298b19c27537fbef1c6bf5765bca7f23ed911c007e6ff3285ea1b8f0"} Nov 01 22:29:31.825056 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.825030 5778 generic.go:356] "Generic (PLEG): container finished" podID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerID="d6ae02e3428643617320ce4bcb02f3c5d30f56c20926a6ec45f6bd7bdc893df9" exitCode=0 Nov 01 22:29:31.825158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:31.825076 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" event={"ID":"7a2e1710-8428-4634-bf53-89efdbfbe2ed","Type":"ContainerDied","Data":"d6ae02e3428643617320ce4bcb02f3c5d30f56c20926a6ec45f6bd7bdc893df9"} Nov 01 22:29:33.850265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:33.850036 5778 generic.go:356] "Generic (PLEG): container finished" podID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerID="682637d514d63794a5dab7ea3187192f9ee04d9f5469e0380a763f49d33f7794" exitCode=0 Nov 01 22:29:33.850265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:33.850147 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" event={"ID":"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a","Type":"ContainerDied","Data":"682637d514d63794a5dab7ea3187192f9ee04d9f5469e0380a763f49d33f7794"} Nov 01 22:29:35.866543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:35.866504 5778 generic.go:356] "Generic (PLEG): container finished" podID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerID="d7e93f4988da5a095351a8d11f8f1b8cd1a0f15213c732105e05b1fd933cfa3d" exitCode=0 Nov 01 22:29:35.867040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:35.866567 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" event={"ID":"f176af06-3cf2-44bd-85fa-e1aabe357631","Type":"ContainerDied","Data":"d7e93f4988da5a095351a8d11f8f1b8cd1a0f15213c732105e05b1fd933cfa3d"} Nov 01 22:29:45.968584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.968545 5778 generic.go:356] "Generic (PLEG): container finished" podID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerID="206d21e012382fd25900e8c3007ecbbc1729c1a9cafa73775ebce6e56bf15167" exitCode=0 Nov 01 22:29:45.969051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.968586 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" event={"ID":"f176af06-3cf2-44bd-85fa-e1aabe357631","Type":"ContainerDied","Data":"206d21e012382fd25900e8c3007ecbbc1729c1a9cafa73775ebce6e56bf15167"} Nov 01 22:29:45.971493 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.971460 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerStarted","Data":"09d8d01509b6f7245fab2ae2d7db8756c491f7b4e4d6fe66c73049701c090ac7"} Nov 01 22:29:45.974091 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.974067 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerID="36b9d079af9976bf243b8966cfaa756bb996ae091d7d35cff1d848b533b9fe12" exitCode=0 Nov 01 22:29:45.974221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.974094 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" event={"ID":"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9","Type":"ContainerDied","Data":"36b9d079af9976bf243b8966cfaa756bb996ae091d7d35cff1d848b533b9fe12"} Nov 01 22:29:45.976476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.976450 5778 generic.go:356] "Generic (PLEG): container finished" podID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerID="17adc53f78b3d3bc90cb15a3a9d86e1e53f7090b04001b34a65372b715f6c45d" exitCode=0 Nov 01 22:29:45.976596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.976526 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" event={"ID":"7a2e1710-8428-4634-bf53-89efdbfbe2ed","Type":"ContainerDied","Data":"17adc53f78b3d3bc90cb15a3a9d86e1e53f7090b04001b34a65372b715f6c45d"} Nov 01 22:29:45.980004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.979978 5778 generic.go:356] "Generic (PLEG): container finished" podID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerID="711252c2f3be06d6ddb38824fff84b7da998cf1f4809d3b7fa659346d3e3be5a" exitCode=0 Nov 01 22:29:45.980124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.980094 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" event={"ID":"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a","Type":"ContainerDied","Data":"711252c2f3be06d6ddb38824fff84b7da998cf1f4809d3b7fa659346d3e3be5a"} Nov 01 22:29:45.983366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.983342 5778 generic.go:356] "Generic (PLEG): container finished" podID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerID="01f527658100f4a2e92dd41bbbbd93b5c917c6d1253971abacd096b698c5caa1" exitCode=0 Nov 01 22:29:45.983479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:45.983389 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" event={"ID":"efe9c226-fa93-4fc8-b0bd-95a33379b743","Type":"ContainerDied","Data":"01f527658100f4a2e92dd41bbbbd93b5c917c6d1253971abacd096b698c5caa1"} Nov 01 22:29:46.008734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:46.008682 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" podStartSLOduration=2.377061888 podStartE2EDuration="32.0086678s" totalImagesPullingTime="29.631605912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:46.007948649 +0000 UTC m=+355.783935479" watchObservedRunningTime="2026-11-01 22:29:46.0086678 +0000 UTC m=+355.784654630" Nov 01 22:29:46.989380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:46.989344 5778 generic.go:356] "Generic (PLEG): container finished" podID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerID="3fa7d9a5c9e1f903b0c8054871f24345e87d18fe44fe8e00511c1b2a4dfa9d7a" exitCode=0 Nov 01 22:29:46.989828 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:46.989442 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" event={"ID":"08b772bd-d151-42b6-a2f5-7b93e1434970","Type":"ContainerDied","Data":"3fa7d9a5c9e1f903b0c8054871f24345e87d18fe44fe8e00511c1b2a4dfa9d7a"} Nov 01 22:29:46.991494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:46.991472 5778 generic.go:356] "Generic (PLEG): container finished" podID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerID="09d8d01509b6f7245fab2ae2d7db8756c491f7b4e4d6fe66c73049701c090ac7" exitCode=0 Nov 01 22:29:46.991631 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:46.991551 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerDied","Data":"09d8d01509b6f7245fab2ae2d7db8756c491f7b4e4d6fe66c73049701c090ac7"} Nov 01 22:29:47.148157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.148134 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:47.164426 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.164397 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\") pod \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " Nov 01 22:29:47.165196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.165161 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle" pod "7a2e1710-8428-4634-bf53-89efdbfbe2ed" (UID: "7a2e1710-8428-4634-bf53-89efdbfbe2ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.176067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.176027 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\" (UniqueName: \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\") pod \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " Nov 01 22:29:47.176067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.176062 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\") pod \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\" (UID: \"7a2e1710-8428-4634-bf53-89efdbfbe2ed\") " Nov 01 22:29:47.176370 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.176353 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.179373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.179332 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj" pod "7a2e1710-8428-4634-bf53-89efdbfbe2ed" (UID: "7a2e1710-8428-4634-bf53-89efdbfbe2ed"). InnerVolumeSpecName "kube-api-access-g2xfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:47.183979 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.183950 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util" pod "7a2e1710-8428-4634-bf53-89efdbfbe2ed" (UID: "7a2e1710-8428-4634-bf53-89efdbfbe2ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.208713 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.208684 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:47.211267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.211243 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:47.220213 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.220194 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:47.230200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.230179 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:47.276690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.276578 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\" (UniqueName: \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\") pod \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " Nov 01 22:29:47.276690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.276643 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\" (UniqueName: \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\") pod \"f176af06-3cf2-44bd-85fa-e1aabe357631\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " Nov 01 22:29:47.276961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.276706 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\") pod \"f176af06-3cf2-44bd-85fa-e1aabe357631\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " Nov 01 22:29:47.276961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.276737 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\") pod \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " Nov 01 22:29:47.276961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.276773 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\" (UniqueName: \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\") pod \"efe9c226-fa93-4fc8-b0bd-95a33379b743\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " Nov 01 22:29:47.277698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.277431 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\") pod \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " Nov 01 22:29:47.277698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.277499 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\" (UniqueName: \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\") pod \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " Nov 01 22:29:47.277698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.277559 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\") pod \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\" (UID: \"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9\") " Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.278016 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle" pod "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" (UID: "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279053 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle" pod "f176af06-3cf2-44bd-85fa-e1aabe357631" (UID: "f176af06-3cf2-44bd-85fa-e1aabe357631"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279131 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle" pod "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" (UID: "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279217 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\") pod \"f176af06-3cf2-44bd-85fa-e1aabe357631\" (UID: \"f176af06-3cf2-44bd-85fa-e1aabe357631\") " Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279305 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\") pod \"efe9c226-fa93-4fc8-b0bd-95a33379b743\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279415 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\") pod \"efe9c226-fa93-4fc8-b0bd-95a33379b743\" (UID: \"efe9c226-fa93-4fc8-b0bd-95a33379b743\") " Nov 01 22:29:47.279616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279446 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\") pod \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\" (UID: \"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a\") " Nov 01 22:29:47.280219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279744 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.280219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279766 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.280219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279781 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xfj\" (UniqueName: \"kubernetes.io/projected/7a2e1710-8428-4634-bf53-89efdbfbe2ed-kube-api-access-g2xfj\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.280219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279798 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2e1710-8428-4634-bf53-89efdbfbe2ed-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.280219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.279812 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.282137 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.282102 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle" pod "efe9c226-fa93-4fc8-b0bd-95a33379b743" (UID: "efe9c226-fa93-4fc8-b0bd-95a33379b743"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.284235 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.284210 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm" pod "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" (UID: "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a"). InnerVolumeSpecName "kube-api-access-5hgtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:47.284476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.284454 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8" pod "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" (UID: "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9"). InnerVolumeSpecName "kube-api-access-d8wg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:47.284729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.284688 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw" pod "efe9c226-fa93-4fc8-b0bd-95a33379b743" (UID: "efe9c226-fa93-4fc8-b0bd-95a33379b743"). InnerVolumeSpecName "kube-api-access-dd5jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:47.286600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.286420 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg" pod "f176af06-3cf2-44bd-85fa-e1aabe357631" (UID: "f176af06-3cf2-44bd-85fa-e1aabe357631"). InnerVolumeSpecName "kube-api-access-qxvtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:47.286600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.286559 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util" pod "f176af06-3cf2-44bd-85fa-e1aabe357631" (UID: "f176af06-3cf2-44bd-85fa-e1aabe357631"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.287148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.287121 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util" pod "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" (UID: "d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.288512 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.288491 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util" pod "efe9c226-fa93-4fc8-b0bd-95a33379b743" (UID: "efe9c226-fa93-4fc8-b0bd-95a33379b743"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.289474 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.289454 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util" pod "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" (UID: "0f2e05e8-a7db-4cb4-99a5-21dcb69d077a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:47.381233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381195 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wg8\" (UniqueName: \"kubernetes.io/projected/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-kube-api-access-d8wg8\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381227 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381240 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381248 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe9c226-fa93-4fc8-b0bd-95a33379b743-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381257 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381264 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hgtm\" (UniqueName: \"kubernetes.io/projected/0f2e05e8-a7db-4cb4-99a5-21dcb69d077a-kube-api-access-5hgtm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381273 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvtg\" (UniqueName: \"kubernetes.io/projected/f176af06-3cf2-44bd-85fa-e1aabe357631-kube-api-access-qxvtg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381282 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f176af06-3cf2-44bd-85fa-e1aabe357631-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.381449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.381290 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd5jw\" (UniqueName: \"kubernetes.io/projected/efe9c226-fa93-4fc8-b0bd-95a33379b743-kube-api-access-dd5jw\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:47.998281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.998253 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" Nov 01 22:29:47.998281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.998258 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tbtrk" event={"ID":"f176af06-3cf2-44bd-85fa-e1aabe357631","Type":"ContainerDied","Data":"e7a6699edaa1b8dfe84a63031a7320957dc2b93553655ac32d60752f85d358a0"} Nov 01 22:29:47.998822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:47.998299 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a6699edaa1b8dfe84a63031a7320957dc2b93553655ac32d60752f85d358a0" Nov 01 22:29:48.000274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.000254 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" Nov 01 22:29:48.000394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.000270 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fbldn" event={"ID":"d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9","Type":"ContainerDied","Data":"205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951"} Nov 01 22:29:48.000394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.000308 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205524a71ffe365f22c8bd355abde31611293d06a91a67053189251916e4d951" Nov 01 22:29:48.002348 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.002330 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" Nov 01 22:29:48.002348 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.002341 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahndjb" event={"ID":"7a2e1710-8428-4634-bf53-89efdbfbe2ed","Type":"ContainerDied","Data":"bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef"} Nov 01 22:29:48.002530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.002378 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd706e9e7343f6e435d68849861fde625ba3f7f4b033f1817e167a43366bcbef" Nov 01 22:29:48.004489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.004471 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" Nov 01 22:29:48.004580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.004488 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwktwn" event={"ID":"0f2e05e8-a7db-4cb4-99a5-21dcb69d077a","Type":"ContainerDied","Data":"64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9"} Nov 01 22:29:48.004580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.004523 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a0117d74635711d06f825f3dae72b406e3db2de078368b731869f2a9eb1cb9" Nov 01 22:29:48.007748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.007665 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" Nov 01 22:29:48.011304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.011273 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d2bpt" event={"ID":"efe9c226-fa93-4fc8-b0bd-95a33379b743","Type":"ContainerDied","Data":"0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5"} Nov 01 22:29:48.011409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.011352 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4ed5024e3d94c5daab7372dd99fbad2c5bfd3f3bc4375d97516a41650236b5" Nov 01 22:29:48.160892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.160841 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:48.182802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.182780 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:48.186470 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.186437 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\") pod \"08b772bd-d151-42b6-a2f5-7b93e1434970\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " Nov 01 22:29:48.186613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.186547 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\") pod \"08b772bd-d151-42b6-a2f5-7b93e1434970\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " Nov 01 22:29:48.186613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.186584 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\" (UniqueName: \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\") pod \"08b772bd-d151-42b6-a2f5-7b93e1434970\" (UID: \"08b772bd-d151-42b6-a2f5-7b93e1434970\") " Nov 01 22:29:48.187240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.187073 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle" pod "08b772bd-d151-42b6-a2f5-7b93e1434970" (UID: "08b772bd-d151-42b6-a2f5-7b93e1434970"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:48.189659 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.189637 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw" pod "08b772bd-d151-42b6-a2f5-7b93e1434970" (UID: "08b772bd-d151-42b6-a2f5-7b93e1434970"). InnerVolumeSpecName "kube-api-access-gt6xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:48.192169 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.192143 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util" pod "08b772bd-d151-42b6-a2f5-7b93e1434970" (UID: "08b772bd-d151-42b6-a2f5-7b93e1434970"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:48.287340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287258 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\" (UniqueName: \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\") pod \"1ae16d4a-b3e1-4b50-988d-69270b4584da\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " Nov 01 22:29:48.287340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287297 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\") pod \"1ae16d4a-b3e1-4b50-988d-69270b4584da\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " Nov 01 22:29:48.287562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287348 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\") pod \"1ae16d4a-b3e1-4b50-988d-69270b4584da\" (UID: \"1ae16d4a-b3e1-4b50-988d-69270b4584da\") " Nov 01 22:29:48.287562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287542 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.287562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287560 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b772bd-d151-42b6-a2f5-7b93e1434970-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.287729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287573 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6xw\" (UniqueName: \"kubernetes.io/projected/08b772bd-d151-42b6-a2f5-7b93e1434970-kube-api-access-gt6xw\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.288001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.287974 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle" pod "1ae16d4a-b3e1-4b50-988d-69270b4584da" (UID: "1ae16d4a-b3e1-4b50-988d-69270b4584da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:48.289597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.289576 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68" pod "1ae16d4a-b3e1-4b50-988d-69270b4584da" (UID: "1ae16d4a-b3e1-4b50-988d-69270b4584da"). InnerVolumeSpecName "kube-api-access-tjh68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:48.292753 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.292732 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util" pod "1ae16d4a-b3e1-4b50-988d-69270b4584da" (UID: "1ae16d4a-b3e1-4b50-988d-69270b4584da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:48.388750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.388724 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjh68\" (UniqueName: \"kubernetes.io/projected/1ae16d4a-b3e1-4b50-988d-69270b4584da-kube-api-access-tjh68\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.388750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.388745 5778 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-util\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.388917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:48.388755 5778 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae16d4a-b3e1-4b50-988d-69270b4584da-bundle\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:48.772242 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:29:48.772204 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:29:49.013576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.013544 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" Nov 01 22:29:49.014117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.013549 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fvksp" event={"ID":"08b772bd-d151-42b6-a2f5-7b93e1434970","Type":"ContainerDied","Data":"777b82175ba1dff58b18d4311c9ae1f5a0f7e95fdf1ef87fa1a0a52764030faa"} Nov 01 22:29:49.014117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.013670 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777b82175ba1dff58b18d4311c9ae1f5a0f7e95fdf1ef87fa1a0a52764030faa" Nov 01 22:29:49.015435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.015409 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" event={"ID":"1ae16d4a-b3e1-4b50-988d-69270b4584da","Type":"ContainerDied","Data":"f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2"} Nov 01 22:29:49.015435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.015428 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw5mbg" Nov 01 22:29:49.015638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:49.015451 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80b4fbe0225d5f610bf38f9539605ad667406c4c057f9208308965a7966cfe2" Nov 01 22:29:51.759984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.759948 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69"] Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760284 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760296 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760310 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760315 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760323 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760327 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760335 5778 cpu_manager.go:414] "containerMap: removing container" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760341 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760352 5778 cpu_manager.go:414] "containerMap: removing container" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760356 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760363 5778 cpu_manager.go:414] "containerMap: removing container" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760367 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="util" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760374 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760379 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760387 5778 cpu_manager.go:414] "containerMap: removing container" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760391 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760396 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760400 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="extract" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760408 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="pull" Nov 01 22:29:51.760405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760412 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760420 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760424 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760430 5778 cpu_manager.go:414] "containerMap: removing container" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760434 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760440 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760444 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760450 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760454 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760461 5778 cpu_manager.go:414] "containerMap: removing container" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760465 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760470 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760473 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760479 5778 cpu_manager.go:414] "containerMap: removing container" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760483 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760489 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760492 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760500 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760504 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="util" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760509 5778 cpu_manager.go:414] "containerMap: removing container" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760513 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="pull" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760519 5778 cpu_manager.go:414] "containerMap: removing container" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760523 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760569 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b772bd-d151-42b6-a2f5-7b93e1434970" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760578 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae16d4a-b3e1-4b50-988d-69270b4584da" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760584 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f176af06-3cf2-44bd-85fa-e1aabe357631" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760590 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fd497b-6f87-4b1b-bea8-7ce004ddf8a9" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760595 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2e05e8-a7db-4cb4-99a5-21dcb69d077a" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760600 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe9c226-fa93-4fc8-b0bd-95a33379b743" containerName="extract" Nov 01 22:29:51.761083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.760605 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e1710-8428-4634-bf53-89efdbfbe2ed" containerName="extract" Nov 01 22:29:51.796229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.796196 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69"] Nov 01 22:29:51.796388 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.796297 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:51.800748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.800726 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:29:51.800748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.800748 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dstps\"" Nov 01 22:29:51.801403 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.801382 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 22:29:51.917466 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.917427 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b94b51-6dc3-46ec-933c-1237b7511ae4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:51.917657 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:51.917504 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pch\" (UniqueName: \"kubernetes.io/projected/82b94b51-6dc3-46ec-933c-1237b7511ae4-kube-api-access-k9pch\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.018950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.018844 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b94b51-6dc3-46ec-933c-1237b7511ae4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.018950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.018923 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pch\" (UniqueName: \"kubernetes.io/projected/82b94b51-6dc3-46ec-933c-1237b7511ae4-kube-api-access-k9pch\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.019214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.019194 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b94b51-6dc3-46ec-933c-1237b7511ae4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.028128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.028101 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pch\" (UniqueName: \"kubernetes.io/projected/82b94b51-6dc3-46ec-933c-1237b7511ae4-kube-api-access-k9pch\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kmd69\" (UID: \"82b94b51-6dc3-46ec-933c-1237b7511ae4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.127214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.127186 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" Nov 01 22:29:52.428840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:52.428800 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69"] Nov 01 22:29:52.433257 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:29:52.433230 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b94b51_6dc3_46ec_933c_1237b7511ae4.slice/crio-c6c33295384825896b2886dfddabacc30cefaba3ea84375c1bff693f1dd2d1d8 WatchSource:0}: Error finding container c6c33295384825896b2886dfddabacc30cefaba3ea84375c1bff693f1dd2d1d8: Status 404 returned error can't find the container with id c6c33295384825896b2886dfddabacc30cefaba3ea84375c1bff693f1dd2d1d8 Nov 01 22:29:53.033880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:53.033828 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" event={"ID":"82b94b51-6dc3-46ec-933c-1237b7511ae4","Type":"ContainerStarted","Data":"c6c33295384825896b2886dfddabacc30cefaba3ea84375c1bff693f1dd2d1d8"} Nov 01 22:29:55.045928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:55.045890 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" event={"ID":"82b94b51-6dc3-46ec-933c-1237b7511ae4","Type":"ContainerStarted","Data":"c9263cd50922fd065d9107f41bb24970c9f0b24525d17ff5f6b70ce89ab1a308"} Nov 01 22:29:55.068063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:29:55.068003 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kmd69" podStartSLOduration=2.067132053 podStartE2EDuration="4.067989885s" totalImagesPullingTime="2.000857832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:55.067438029 +0000 UTC m=+364.843424882" watchObservedRunningTime="2026-11-01 22:29:55.067989885 +0000 UTC m=+364.843976714" Nov 01 22:30:01.405784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.405702 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s"] Nov 01 22:30:01.408967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.408951 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:01.412897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.412874 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:01.415546 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.415529 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:01.417257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.417242 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-59c9n\"" Nov 01 22:30:01.420402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.420376 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s"] Nov 01 22:30:01.502905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.502845 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbqz\" (UniqueName: \"kubernetes.io/projected/66266fef-815a-475c-a38f-e5e8de8fa883-kube-api-access-lrbqz\") pod \"limitador-operator-controller-manager-5778bb4449-62v9s\" (UID: \"66266fef-815a-475c-a38f-e5e8de8fa883\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:01.603804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.603766 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbqz\" (UniqueName: \"kubernetes.io/projected/66266fef-815a-475c-a38f-e5e8de8fa883-kube-api-access-lrbqz\") pod \"limitador-operator-controller-manager-5778bb4449-62v9s\" (UID: \"66266fef-815a-475c-a38f-e5e8de8fa883\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:01.617272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.617239 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbqz\" (UniqueName: \"kubernetes.io/projected/66266fef-815a-475c-a38f-e5e8de8fa883-kube-api-access-lrbqz\") pod \"limitador-operator-controller-manager-5778bb4449-62v9s\" (UID: \"66266fef-815a-475c-a38f-e5e8de8fa883\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:01.722210 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.722118 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:01.996293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:01.996265 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s"] Nov 01 22:30:01.999010 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:01.998981 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66266fef_815a_475c_a38f_e5e8de8fa883.slice/crio-47057ddf8a76c920be57caae47c5c775a996f8ed431d0f291128d30fa7288f30 WatchSource:0}: Error finding container 47057ddf8a76c920be57caae47c5c775a996f8ed431d0f291128d30fa7288f30: Status 404 returned error can't find the container with id 47057ddf8a76c920be57caae47c5c775a996f8ed431d0f291128d30fa7288f30 Nov 01 22:30:02.082702 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:02.082661 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" event={"ID":"66266fef-815a-475c-a38f-e5e8de8fa883","Type":"ContainerStarted","Data":"47057ddf8a76c920be57caae47c5c775a996f8ed431d0f291128d30fa7288f30"} Nov 01 22:30:06.214018 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.213981 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hbg9l"] Nov 01 22:30:06.217532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.217513 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.227966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.227939 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 22:30:06.228104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.227980 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hvtp7\"" Nov 01 22:30:06.228104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.228036 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 22:30:06.229167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.229141 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hbg9l"] Nov 01 22:30:06.344545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.344507 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsz5\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-kube-api-access-zhsz5\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.344734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.344565 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-bound-sa-token\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.445936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.445903 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsz5\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-kube-api-access-zhsz5\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.446143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.445960 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-bound-sa-token\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.454405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.454381 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-bound-sa-token\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.454533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.454423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsz5\" (UniqueName: \"kubernetes.io/projected/ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa-kube-api-access-zhsz5\") pod \"cert-manager-597b95cb84-hbg9l\" (UID: \"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa\") " pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.541613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.541529 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hbg9l" Nov 01 22:30:06.816839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:06.816812 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hbg9l"] Nov 01 22:30:06.819780 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:06.819748 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd24f4b_9c31_485e_8961_e5d3e8d6d0aa.slice/crio-3c1cd17c274e2e555873f6deca50509769f05042e1af6e00382fdf48dc73409b WatchSource:0}: Error finding container 3c1cd17c274e2e555873f6deca50509769f05042e1af6e00382fdf48dc73409b: Status 404 returned error can't find the container with id 3c1cd17c274e2e555873f6deca50509769f05042e1af6e00382fdf48dc73409b Nov 01 22:30:07.109885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:07.109776 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hbg9l" event={"ID":"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa","Type":"ContainerStarted","Data":"3c1cd17c274e2e555873f6deca50509769f05042e1af6e00382fdf48dc73409b"} Nov 01 22:30:07.111487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:07.111457 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" event={"ID":"66266fef-815a-475c-a38f-e5e8de8fa883","Type":"ContainerStarted","Data":"f513b61821181150d691edca1dac3d81e2170e14a57cc1fb7b8f385d5a296c12"} Nov 01 22:30:07.111623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:07.111584 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:07.131977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:07.131933 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" podStartSLOduration=1.5049571130000001 podStartE2EDuration="6.131920458s" totalImagesPullingTime="4.626963345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:07.130709302 +0000 UTC m=+376.906696130" watchObservedRunningTime="2026-11-01 22:30:07.131920458 +0000 UTC m=+376.907907287" Nov 01 22:30:09.045159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.045119 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4"] Nov 01 22:30:09.049318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.049290 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.058679 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.058656 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 22:30:09.058815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.058727 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 22:30:09.058910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.058866 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:09.059058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.058941 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:09.059058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.059031 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bqmh4\"" Nov 01 22:30:09.061099 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.061077 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 22:30:09.075114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.075085 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4"] Nov 01 22:30:09.168514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168472 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.168710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168541 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.168710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168630 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.168710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168665 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7mz\" (UniqueName: \"kubernetes.io/projected/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-kube-api-access-vq7mz\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.168922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168832 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.168922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.168886 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.269742 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.269707 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.269742 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.269756 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.270035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.269910 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.270035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.269990 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.270155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.270044 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.270155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.270077 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7mz\" (UniqueName: \"kubernetes.io/projected/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-kube-api-access-vq7mz\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.270259 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.270189 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.272710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.272683 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.273053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.273028 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.273189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.273147 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.273495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.273469 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.279745 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.279723 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7mz\" (UniqueName: \"kubernetes.io/projected/ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4-kube-api-access-vq7mz\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-dbtw4\" (UID: \"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.367587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.367497 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:09.674408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:09.674379 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4"] Nov 01 22:30:09.676031 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:09.676006 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfe1a4a_beb6_41c0_9f3f_9c4e8588b3e4.slice/crio-0e214bfe87670a52c394ba1e7d9370b00782250eb2832ab442c46c04c24014e3 WatchSource:0}: Error finding container 0e214bfe87670a52c394ba1e7d9370b00782250eb2832ab442c46c04c24014e3: Status 404 returned error can't find the container with id 0e214bfe87670a52c394ba1e7d9370b00782250eb2832ab442c46c04c24014e3 Nov 01 22:30:10.128932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.128889 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hbg9l" event={"ID":"ccd24f4b-9c31-485e-8961-e5d3e8d6d0aa","Type":"ContainerStarted","Data":"dff1997bc6fd7dbdbf72fdad43a42278bb4cc21030acc1bb973fe09a2e13599c"} Nov 01 22:30:10.130340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.130313 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" event={"ID":"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4","Type":"ContainerStarted","Data":"0e214bfe87670a52c394ba1e7d9370b00782250eb2832ab442c46c04c24014e3"} Nov 01 22:30:10.151795 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.151750 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hbg9l" podStartSLOduration=1.3869606110000001 podStartE2EDuration="4.151735151s" totalImagesPullingTime="2.76477454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:10.150766176 +0000 UTC m=+379.926753007" watchObservedRunningTime="2026-11-01 22:30:10.151735151 +0000 UTC m=+379.927721979" Nov 01 22:30:10.356949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.356911 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:10.360265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.360242 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.373002 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.372974 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lvjbk\"" Nov 01 22:30:10.378028 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.378005 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:10.479441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.479403 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.479622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.479451 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh7l\" (UniqueName: \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.580036 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.580005 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.580214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.580050 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh7l\" (UniqueName: \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.580456 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.580427 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.592477 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.592455 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh7l\" (UniqueName: \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\") pod \"kuadrant-operator-controller-manager-cf8469d7b-zmzdb\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.673926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.673890 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:10.970410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:10.970334 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:10.973836 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:10.973807 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dd3ea8_2b33_43bc_a426_8a284a7470b1.slice/crio-d165a282ca0f49492ffab04fcd6b66751ab00048f4aea09a8c08651b14fbd669 WatchSource:0}: Error finding container d165a282ca0f49492ffab04fcd6b66751ab00048f4aea09a8c08651b14fbd669: Status 404 returned error can't find the container with id d165a282ca0f49492ffab04fcd6b66751ab00048f4aea09a8c08651b14fbd669 Nov 01 22:30:11.139272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:11.139173 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" event={"ID":"d1dd3ea8-2b33-43bc-a426-8a284a7470b1","Type":"ContainerStarted","Data":"d165a282ca0f49492ffab04fcd6b66751ab00048f4aea09a8c08651b14fbd669"} Nov 01 22:30:14.549385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.549352 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m"] Nov 01 22:30:14.553053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.553031 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:14.559655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.559633 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wx4zt\"" Nov 01 22:30:14.566821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.566776 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 22:30:14.568672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.568652 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m"] Nov 01 22:30:14.719845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.719807 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgns7\" (UniqueName: \"kubernetes.io/projected/cd99f908-504c-4f19-b77d-a8add7cd065b-kube-api-access-cgns7\") pod \"dns-operator-controller-manager-8557cbdb84-7g97m\" (UID: \"cd99f908-504c-4f19-b77d-a8add7cd065b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:14.820587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.820501 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgns7\" (UniqueName: \"kubernetes.io/projected/cd99f908-504c-4f19-b77d-a8add7cd065b-kube-api-access-cgns7\") pod \"dns-operator-controller-manager-8557cbdb84-7g97m\" (UID: \"cd99f908-504c-4f19-b77d-a8add7cd065b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:14.831057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.830638 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgns7\" (UniqueName: \"kubernetes.io/projected/cd99f908-504c-4f19-b77d-a8add7cd065b-kube-api-access-cgns7\") pod \"dns-operator-controller-manager-8557cbdb84-7g97m\" (UID: \"cd99f908-504c-4f19-b77d-a8add7cd065b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:14.869999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:14.869965 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:15.153919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:15.153761 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m"] Nov 01 22:30:15.157350 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:15.157319 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd99f908_504c_4f19_b77d_a8add7cd065b.slice/crio-b347edd05fec34b553b3494b1e19cccede7bba26d5fc57291e90299cea9c33ec WatchSource:0}: Error finding container b347edd05fec34b553b3494b1e19cccede7bba26d5fc57291e90299cea9c33ec: Status 404 returned error can't find the container with id b347edd05fec34b553b3494b1e19cccede7bba26d5fc57291e90299cea9c33ec Nov 01 22:30:16.168353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:16.168274 5778 generic.go:356] "Generic (PLEG): container finished" podID="ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4" containerID="91a7974acfd2631c6cae3bb96e34258555117572b8143aa1ebe670e0606d24a1" exitCode=0 Nov 01 22:30:16.168778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:16.168367 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" event={"ID":"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4","Type":"ContainerDied","Data":"91a7974acfd2631c6cae3bb96e34258555117572b8143aa1ebe670e0606d24a1"} Nov 01 22:30:16.170048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:16.169879 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" event={"ID":"cd99f908-504c-4f19-b77d-a8add7cd065b","Type":"ContainerStarted","Data":"b347edd05fec34b553b3494b1e19cccede7bba26d5fc57291e90299cea9c33ec"} Nov 01 22:30:17.176138 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:17.176101 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" event={"ID":"ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4","Type":"ContainerStarted","Data":"d58b81313123d2311c57d6540f025b9cde14d7f4f6d830d7c0230a99862be7e1"} Nov 01 22:30:17.176568 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:17.176201 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:17.219353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:17.219294 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" podStartSLOduration=1.600852374 podStartE2EDuration="8.219279311s" totalImagesPullingTime="6.068818913s" totalInitContainerRuntime="549.608024ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:17.216156174 +0000 UTC m=+386.992143004" watchObservedRunningTime="2026-11-01 22:30:17.219279311 +0000 UTC m=+386.995266138" Nov 01 22:30:18.119658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.119625 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-62v9s" Nov 01 22:30:18.313982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.313946 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq"] Nov 01 22:30:18.317269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.317250 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:18.328308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.328278 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lf9wx\"" Nov 01 22:30:18.329006 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.328980 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq"] Nov 01 22:30:18.450356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.450321 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2tt\" (UniqueName: \"kubernetes.io/projected/2fec942a-c9c0-424d-826d-5a03f4aa9a5d-kube-api-access-ws2tt\") pod \"authorino-operator-7b7bb4bc77-tlfjq\" (UID: \"2fec942a-c9c0-424d-826d-5a03f4aa9a5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:18.551186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.551151 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2tt\" (UniqueName: \"kubernetes.io/projected/2fec942a-c9c0-424d-826d-5a03f4aa9a5d-kube-api-access-ws2tt\") pod \"authorino-operator-7b7bb4bc77-tlfjq\" (UID: \"2fec942a-c9c0-424d-826d-5a03f4aa9a5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:18.560807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.560776 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2tt\" (UniqueName: \"kubernetes.io/projected/2fec942a-c9c0-424d-826d-5a03f4aa9a5d-kube-api-access-ws2tt\") pod \"authorino-operator-7b7bb4bc77-tlfjq\" (UID: \"2fec942a-c9c0-424d-826d-5a03f4aa9a5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:18.630390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.630359 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:18.774953 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:30:18.774840 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:30:18.907218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:18.907183 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq"] Nov 01 22:30:18.910035 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:18.910009 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fec942a_c9c0_424d_826d_5a03f4aa9a5d.slice/crio-22254b9fe891b089e1cad0e1cc901da574b1b4b4510a606584d8f96bbf679772 WatchSource:0}: Error finding container 22254b9fe891b089e1cad0e1cc901da574b1b4b4510a606584d8f96bbf679772: Status 404 returned error can't find the container with id 22254b9fe891b089e1cad0e1cc901da574b1b4b4510a606584d8f96bbf679772 Nov 01 22:30:19.189097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:19.189069 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" event={"ID":"2fec942a-c9c0-424d-826d-5a03f4aa9a5d","Type":"ContainerStarted","Data":"22254b9fe891b089e1cad0e1cc901da574b1b4b4510a606584d8f96bbf679772"} Nov 01 22:30:21.201458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:21.201418 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" event={"ID":"cd99f908-504c-4f19-b77d-a8add7cd065b","Type":"ContainerStarted","Data":"6cfc517a61313f3a9c37c15a4515bcece951be06ca16c66cff4a18e699781762"} Nov 01 22:30:21.201887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:21.201560 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:21.220312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:21.220263 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" podStartSLOduration=2.18438634 podStartE2EDuration="7.220248886s" totalImagesPullingTime="5.035862546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:21.218762081 +0000 UTC m=+390.994748911" watchObservedRunningTime="2026-11-01 22:30:21.220248886 +0000 UTC m=+390.996235715" Nov 01 22:30:23.215349 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:23.215306 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" event={"ID":"d1dd3ea8-2b33-43bc-a426-8a284a7470b1","Type":"ContainerStarted","Data":"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a"} Nov 01 22:30:23.215715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:23.215538 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:23.236166 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:23.236120 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" podStartSLOduration=1.159975212 podStartE2EDuration="13.236104109s" totalImagesPullingTime="12.076128897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:23.234592438 +0000 UTC m=+393.010579276" watchObservedRunningTime="2026-11-01 22:30:23.236104109 +0000 UTC m=+393.012090937" Nov 01 22:30:24.223893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:24.223835 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" event={"ID":"2fec942a-c9c0-424d-826d-5a03f4aa9a5d","Type":"ContainerStarted","Data":"9a126bdae2526f978c2d84f20b8e7621d5f5ebeed8fad87aaf3af83754d1e140"} Nov 01 22:30:24.224323 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:24.223937 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:24.239414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:24.239354 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" podStartSLOduration=1.946533495 podStartE2EDuration="6.23933794s" totalImagesPullingTime="4.292804445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:24.239066783 +0000 UTC m=+394.015053611" watchObservedRunningTime="2026-11-01 22:30:24.23933794 +0000 UTC m=+394.015324771" Nov 01 22:30:28.184518 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:28.184490 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-dbtw4" Nov 01 22:30:32.211254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:32.211217 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7g97m" Nov 01 22:30:34.225978 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.225942 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:34.517815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.517734 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 22:30:34.521362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.521347 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.534564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.534542 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 22:30:34.585219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.585190 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmx2\" (UniqueName: \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.585343 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.585237 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.685986 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.685949 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.686154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.686068 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmx2\" (UniqueName: \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.686417 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.686395 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.695460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.695431 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmx2\" (UniqueName: \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\") pod \"kuadrant-operator-controller-manager-678d87766f-c2f4p\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:34.834089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:34.834006 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:35.115552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:35.115524 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 22:30:35.116982 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:35.116957 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72b62f8_fa79_418a_a5fe_e92fa199498a.slice/crio-69fc897cd4697bb823bcdf790e227c59211c34a32eeaba5381ecd91a0d26993d WatchSource:0}: Error finding container 69fc897cd4697bb823bcdf790e227c59211c34a32eeaba5381ecd91a0d26993d: Status 404 returned error can't find the container with id 69fc897cd4697bb823bcdf790e227c59211c34a32eeaba5381ecd91a0d26993d Nov 01 22:30:35.277890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:35.277844 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerStarted","Data":"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377"} Nov 01 22:30:35.278300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:35.277914 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerStarted","Data":"69fc897cd4697bb823bcdf790e227c59211c34a32eeaba5381ecd91a0d26993d"} Nov 01 22:30:35.278300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:35.277994 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:35.303601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:35.303548 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" podStartSLOduration=1.303530594 podStartE2EDuration="1.303530594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:35.303041375 +0000 UTC m=+405.079028206" watchObservedRunningTime="2026-11-01 22:30:35.303530594 +0000 UTC m=+405.079517426" Nov 01 22:30:36.234822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:36.234791 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlfjq" Nov 01 22:30:40.598133 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.598103 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff"] Nov 01 22:30:40.606538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.606516 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.630746 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.630722 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-d9rvb\"" Nov 01 22:30:40.641284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641260 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641391 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641308 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641391 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641371 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641404 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641426 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zqj\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-kube-api-access-z7zqj\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641495 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641560 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641524 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641560 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641543 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f51c778-961a-495e-9229-2be5253a26bb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.641623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.641560 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f51c778-961a-495e-9229-2be5253a26bb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.652109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.652082 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff"] Nov 01 22:30:40.742380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742334 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742389 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742417 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f51c778-961a-495e-9229-2be5253a26bb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742444 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f51c778-961a-495e-9229-2be5253a26bb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742498 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742561 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742589 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742635 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742659 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zqj\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-kube-api-access-z7zqj\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.742837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.742793 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.743155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.743075 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.743211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.743159 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.743345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.743162 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.743345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.743253 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f51c778-961a-495e-9229-2be5253a26bb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.745142 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.745121 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f51c778-961a-495e-9229-2be5253a26bb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.745550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.745529 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f51c778-961a-495e-9229-2be5253a26bb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.781573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.781546 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.789927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.789830 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zqj\" (UniqueName: \"kubernetes.io/projected/4f51c778-961a-495e-9229-2be5253a26bb-kube-api-access-z7zqj\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5h6rff\" (UID: \"4f51c778-961a-495e-9229-2be5253a26bb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:40.919694 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:40.919648 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:41.219329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.219303 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff"] Nov 01 22:30:41.221117 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:41.221083 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f51c778_961a_495e_9229_2be5253a26bb.slice/crio-efbc261a605494c2ea49b2270163317ee24b5265a0b9f76bad4d06e6a877bf86 WatchSource:0}: Error finding container efbc261a605494c2ea49b2270163317ee24b5265a0b9f76bad4d06e6a877bf86: Status 404 returned error can't find the container with id efbc261a605494c2ea49b2270163317ee24b5265a0b9f76bad4d06e6a877bf86 Nov 01 22:30:41.223484 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.223456 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:41.223564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.223524 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:41.223564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.223560 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:41.313258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.313226 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" event={"ID":"4f51c778-961a-495e-9229-2be5253a26bb","Type":"ContainerStarted","Data":"60173b612ff2f332348d680a759333be9861623eabc62f2a0ebfa6c39d89f485"} Nov 01 22:30:41.313432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.313288 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" event={"ID":"4f51c778-961a-495e-9229-2be5253a26bb","Type":"ContainerStarted","Data":"efbc261a605494c2ea49b2270163317ee24b5265a0b9f76bad4d06e6a877bf86"} Nov 01 22:30:41.345224 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.345171 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" podStartSLOduration=1.34515503 podStartE2EDuration="1.34515503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:41.344143379 +0000 UTC m=+411.120130210" watchObservedRunningTime="2026-11-01 22:30:41.34515503 +0000 UTC m=+411.121141858" Nov 01 22:30:41.920240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:41.920206 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:42.925660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:42.925631 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:43.324568 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:43.324490 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:43.325961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:43.325940 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5h6rff" Nov 01 22:30:44.457596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.457561 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b"] Nov 01 22:30:44.461150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.461131 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.475240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.475201 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-h54td\"" Nov 01 22:30:44.477564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.477538 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b"] Nov 01 22:30:44.508798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.508764 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 22:30:44.508950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.508901 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 22:30:44.512407 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.512387 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 01 22:30:44.577398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.577364 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xqf\" (UniqueName: \"kubernetes.io/projected/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-kube-api-access-44xqf\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.577549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.577419 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.577549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.577526 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.577628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.577585 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679052 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679108 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679170 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679228 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xqf\" (UniqueName: \"kubernetes.io/projected/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-kube-api-access-44xqf\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679441 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.679530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.679479 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.680041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.680024 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.687618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.687594 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xqf\" (UniqueName: \"kubernetes.io/projected/7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac-kube-api-access-44xqf\") pod \"kserve-module-controller-manager-7566cf5fc8-ztp9b\" (UID: \"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:44.773487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:44.773409 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:45.049566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:45.049515 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b"] Nov 01 22:30:45.051875 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:30:45.051824 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f386979_0e3b_4a16_b4e2_a4dc2fe4dbac.slice/crio-36edb5aeda3bc467daba1d5ffe6d0769f40d024671bced2c6441161b86f09bfd WatchSource:0}: Error finding container 36edb5aeda3bc467daba1d5ffe6d0769f40d024671bced2c6441161b86f09bfd: Status 404 returned error can't find the container with id 36edb5aeda3bc467daba1d5ffe6d0769f40d024671bced2c6441161b86f09bfd Nov 01 22:30:45.336250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:45.336163 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" event={"ID":"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac","Type":"ContainerStarted","Data":"36edb5aeda3bc467daba1d5ffe6d0769f40d024671bced2c6441161b86f09bfd"} Nov 01 22:30:46.285276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.285241 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:30:46.351311 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.351223 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:46.352116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.352084 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" podUID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" containerName="manager" containerID="cri-o://2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a" gracePeriod=10 Nov 01 22:30:46.621160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.621136 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:46.698222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.698182 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\") pod \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " Nov 01 22:30:46.698387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.698288 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\" (UniqueName: \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\") pod \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\" (UID: \"d1dd3ea8-2b33-43bc-a426-8a284a7470b1\") " Nov 01 22:30:46.698936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.698904 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume" pod "d1dd3ea8-2b33-43bc-a426-8a284a7470b1" (UID: "d1dd3ea8-2b33-43bc-a426-8a284a7470b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:46.700898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.700872 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l" pod "d1dd3ea8-2b33-43bc-a426-8a284a7470b1" (UID: "d1dd3ea8-2b33-43bc-a426-8a284a7470b1"). InnerVolumeSpecName "kube-api-access-4lh7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:46.799620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.799585 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh7l\" (UniqueName: \"kubernetes.io/projected/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-kube-api-access-4lh7l\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:46.799620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:46.799614 5778 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dd3ea8-2b33-43bc-a426-8a284a7470b1-extensions-socket-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:47.354767 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.354731 5778 generic.go:356] "Generic (PLEG): container finished" podID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" containerID="2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a" exitCode=0 Nov 01 22:30:47.355296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.354790 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" Nov 01 22:30:47.355296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.354826 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" event={"ID":"d1dd3ea8-2b33-43bc-a426-8a284a7470b1","Type":"ContainerDied","Data":"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a"} Nov 01 22:30:47.355296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.354922 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb" event={"ID":"d1dd3ea8-2b33-43bc-a426-8a284a7470b1","Type":"ContainerDied","Data":"d165a282ca0f49492ffab04fcd6b66751ab00048f4aea09a8c08651b14fbd669"} Nov 01 22:30:47.355296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.354956 5778 scope.go:122] "RemoveContainer" containerID="2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a" Nov 01 22:30:47.367722 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.367697 5778 scope.go:122] "RemoveContainer" containerID="2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a" Nov 01 22:30:47.368043 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:30:47.368020 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a\": container with ID starting with 2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a not found: ID does not exist" containerID="2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a" Nov 01 22:30:47.368126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.368053 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a"} err="failed to get container status \"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a\": rpc error: code = NotFound desc = could not find container \"2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a\": container with ID starting with 2e5c7764865edaffab1ae3cd98f91f339410720eca58d1fba90db8bab424338a not found: ID does not exist" Nov 01 22:30:47.397091 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.397060 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:47.430096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:47.430067 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-zmzdb"] Nov 01 22:30:48.372472 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:48.372438 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" path="/var/lib/kubelet/pods/d1dd3ea8-2b33-43bc-a426-8a284a7470b1/volumes" Nov 01 22:30:48.772574 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:30:48.772532 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:30:49.368722 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:49.368681 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" event={"ID":"7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac","Type":"ContainerStarted","Data":"fe26d31707a0a8c5ab8cefea4a6b65f09cb696240d189c685f501ee1389c6caa"} Nov 01 22:30:49.368947 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:49.368813 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:30:49.394483 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:30:49.394430 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" podStartSLOduration=2.046708029 podStartE2EDuration="5.394413179s" totalImagesPullingTime="3.34770515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:49.391993607 +0000 UTC m=+419.167980435" watchObservedRunningTime="2026-11-01 22:30:49.394413179 +0000 UTC m=+419.170400007" Nov 01 22:31:08.453674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.453637 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl"] Nov 01 22:31:08.454202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.454183 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" containerName="manager" Nov 01 22:31:08.454252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.454204 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" containerName="manager" Nov 01 22:31:08.454288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.454267 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1dd3ea8-2b33-43bc-a426-8a284a7470b1" containerName="manager" Nov 01 22:31:08.457522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.457498 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.464368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.464341 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl"] Nov 01 22:31:08.465223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.465029 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:31:08.465223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.465040 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:31:08.559962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.559933 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 22:31:08.559962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.559950 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gnd87\"" Nov 01 22:31:08.604473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.604441 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6hc\" (UniqueName: \"kubernetes.io/projected/51378849-677d-4da1-82f1-f2289c386f77-kube-api-access-wq6hc\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.604635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.604524 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51378849-677d-4da1-82f1-f2289c386f77-data\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.705901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.705792 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51378849-677d-4da1-82f1-f2289c386f77-data\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.705901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.705888 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6hc\" (UniqueName: \"kubernetes.io/projected/51378849-677d-4da1-82f1-f2289c386f77-kube-api-access-wq6hc\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.706237 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.706217 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51378849-677d-4da1-82f1-f2289c386f77-data\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.719114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.719087 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6hc\" (UniqueName: \"kubernetes.io/projected/51378849-677d-4da1-82f1-f2289c386f77-kube-api-access-wq6hc\") pod \"postgres-7cdbf6dc44-c2thl\" (UID: \"51378849-677d-4da1-82f1-f2289c386f77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:08.771988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:08.771963 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:09.044092 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:09.044017 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl"] Nov 01 22:31:09.046945 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:31:09.046910 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51378849_677d_4da1_82f1_f2289c386f77.slice/crio-b15f72832b4f7ab5c50330f1520971e469e3a04fce015e2a61633f55b566a8f8 WatchSource:0}: Error finding container b15f72832b4f7ab5c50330f1520971e469e3a04fce015e2a61633f55b566a8f8: Status 404 returned error can't find the container with id b15f72832b4f7ab5c50330f1520971e469e3a04fce015e2a61633f55b566a8f8 Nov 01 22:31:09.468059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:09.468022 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" event={"ID":"51378849-677d-4da1-82f1-f2289c386f77","Type":"ContainerStarted","Data":"b15f72832b4f7ab5c50330f1520971e469e3a04fce015e2a61633f55b566a8f8"} Nov 01 22:31:10.375971 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:10.375936 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ztp9b" Nov 01 22:31:15.505875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:15.505811 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" event={"ID":"51378849-677d-4da1-82f1-f2289c386f77","Type":"ContainerStarted","Data":"cf4641fec8a794148d7d95aedabcd3dffdfb3f66ac13747559bf1fa3676aace9"} Nov 01 22:31:15.506279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:15.505906 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:15.522488 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:15.522440 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" podStartSLOduration=2.072245838 podStartE2EDuration="7.522426355s" totalImagesPullingTime="5.450180517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.521251322 +0000 UTC m=+445.297238153" watchObservedRunningTime="2026-11-01 22:31:15.522426355 +0000 UTC m=+445.298413183" Nov 01 22:31:18.772988 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:31:18.772951 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:31:21.542989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:21.542963 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c2thl" Nov 01 22:31:33.278421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.278344 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2"] Nov 01 22:31:33.282066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.282044 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.296072 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.296047 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 22:31:33.296256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.296235 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 22:31:33.296610 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.296582 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2"] Nov 01 22:31:33.324125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.324094 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.324268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.324201 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d1d250-8467-498a-87e7-4037d9f66ee2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.324268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.324238 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gqn\" (UniqueName: \"kubernetes.io/projected/13d1d250-8467-498a-87e7-4037d9f66ee2-kube-api-access-x2gqn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.324406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.324385 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.333292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.333269 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 22:31:33.333798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.333782 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-tq2pl\"" Nov 01 22:31:33.425372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.425334 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d1d250-8467-498a-87e7-4037d9f66ee2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.425372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.425368 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gqn\" (UniqueName: \"kubernetes.io/projected/13d1d250-8467-498a-87e7-4037d9f66ee2-kube-api-access-x2gqn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.425743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.425718 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.425826 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.425810 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.426438 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.426413 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.426987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.426966 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13d1d250-8467-498a-87e7-4037d9f66ee2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.428237 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.428204 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d1d250-8467-498a-87e7-4037d9f66ee2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.434809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.434781 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gqn\" (UniqueName: \"kubernetes.io/projected/13d1d250-8467-498a-87e7-4037d9f66ee2-kube-api-access-x2gqn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zpxk2\" (UID: \"13d1d250-8467-498a-87e7-4037d9f66ee2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.598365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.598280 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:33.876946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:33.876918 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2"] Nov 01 22:31:33.879665 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:31:33.879625 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d1d250_8467_498a_87e7_4037d9f66ee2.slice/crio-253b4f827c721b3a312c8a00c2e969bbaca556b95c34d5cd820f227736597e46 WatchSource:0}: Error finding container 253b4f827c721b3a312c8a00c2e969bbaca556b95c34d5cd820f227736597e46: Status 404 returned error can't find the container with id 253b4f827c721b3a312c8a00c2e969bbaca556b95c34d5cd820f227736597e46 Nov 01 22:31:34.614014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:34.613984 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" event={"ID":"13d1d250-8467-498a-87e7-4037d9f66ee2","Type":"ContainerStarted","Data":"253b4f827c721b3a312c8a00c2e969bbaca556b95c34d5cd820f227736597e46"} Nov 01 22:31:39.618100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.618059 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:31:39.624892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.624215 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.639666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.639635 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:31:39.694312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.694267 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.694538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.694510 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-csh2g\"" Nov 01 22:31:39.694538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.694511 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wqq\" (UniqueName: \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.694806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.694788 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 22:31:39.796025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.795990 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wqq\" (UniqueName: \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.796218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.796045 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.798932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.798909 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.820094 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.820037 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wqq\" (UniqueName: \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\") pod \"llmisvc-controller-manager-869d5dcc5d-nrqf6\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:39.949994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:39.949775 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:40.262055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:40.262025 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:31:40.264978 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:31:40.264944 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod060f0e1b_c856_4227_b784_00e8e099f920.slice/crio-9eb4e966c1a4f232e8042369909abce8bd1776fc0b97c6d616a59427ad3deb4f WatchSource:0}: Error finding container 9eb4e966c1a4f232e8042369909abce8bd1776fc0b97c6d616a59427ad3deb4f: Status 404 returned error can't find the container with id 9eb4e966c1a4f232e8042369909abce8bd1776fc0b97c6d616a59427ad3deb4f Nov 01 22:31:40.656572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:40.656534 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" event={"ID":"060f0e1b-c856-4227-b784-00e8e099f920","Type":"ContainerStarted","Data":"9eb4e966c1a4f232e8042369909abce8bd1776fc0b97c6d616a59427ad3deb4f"} Nov 01 22:31:46.695480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:46.695439 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" event={"ID":"060f0e1b-c856-4227-b784-00e8e099f920","Type":"ContainerStarted","Data":"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329"} Nov 01 22:31:46.695997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:46.695618 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:31:46.715335 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:46.715277 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" podStartSLOduration=1.581177736 podStartE2EDuration="7.715256628s" totalImagesPullingTime="6.134078892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:46.712574284 +0000 UTC m=+476.488561114" watchObservedRunningTime="2026-11-01 22:31:46.715256628 +0000 UTC m=+476.491243459" Nov 01 22:31:48.779592 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:31:48.779553 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:31:49.716332 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:49.716295 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" event={"ID":"13d1d250-8467-498a-87e7-4037d9f66ee2","Type":"ContainerStarted","Data":"c7e617000b1a122627f4e73e451cc0a0954a5b8ebb5308846495903a504b9c6b"} Nov 01 22:31:49.716545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:49.716415 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:31:49.734171 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:31:49.734122 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" podStartSLOduration=1.632038135 podStartE2EDuration="16.734100756s" totalImagesPullingTime="15.102062621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:49.732776686 +0000 UTC m=+479.508763515" watchObservedRunningTime="2026-11-01 22:31:49.734100756 +0000 UTC m=+479.510087586" Nov 01 22:32:00.729359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:00.729325 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zpxk2" Nov 01 22:32:15.608713 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.608676 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 22:32:15.612505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.612481 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.622468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.622440 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 22:32:15.643498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.643474 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 22:32:15.643498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.643473 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 22:32:15.643664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.643535 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:32:15.723662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.723624 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.723814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.723684 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxqf\" (UniqueName: \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.723814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.723785 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.723933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.723818 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.824270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.824238 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxqf\" (UniqueName: \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.824445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.824287 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.824445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.824306 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.824445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.824360 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.825045 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.825020 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.827051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.827028 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.827136 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.827064 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.832508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.832480 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxqf\" (UniqueName: \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\") pod \"maas-api-dcc8544c-lbw44\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:15.874895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.874812 5778 generic.go:356] "Generic (PLEG): container finished" podID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerID="857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24" exitCode=0 Nov 01 22:32:15.874895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.874884 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerDied","Data":"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24"} Nov 01 22:32:15.875289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.875272 5778 scope.go:122] "RemoveContainer" containerID="857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24" Nov 01 22:32:15.876491 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.876463 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:32:15.876557 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.876539 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:32:15.876592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.876570 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:32:15.927501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:15.927477 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:16.221088 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:16.221054 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 22:32:16.223014 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:32:16.222983 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1939254_2eef_4572_a8e1_6d075ab39eb9.slice/crio-978fa81dfd8f4de7cc36dc9b32de5da7f246c962d952d3497a77bc69991a9613 WatchSource:0}: Error finding container 978fa81dfd8f4de7cc36dc9b32de5da7f246c962d952d3497a77bc69991a9613: Status 404 returned error can't find the container with id 978fa81dfd8f4de7cc36dc9b32de5da7f246c962d952d3497a77bc69991a9613 Nov 01 22:32:16.882787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:16.882746 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" event={"ID":"b1939254-2eef-4572-a8e1-6d075ab39eb9","Type":"ContainerStarted","Data":"978fa81dfd8f4de7cc36dc9b32de5da7f246c962d952d3497a77bc69991a9613"} Nov 01 22:32:16.884949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:16.884924 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerStarted","Data":"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884"} Nov 01 22:32:17.039876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:17.037286 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:32:17.039876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:17.037420 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Nov 01 22:32:17.039876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:17.037469 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Nov 01 22:32:17.706447 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:17.706420 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:32:18.039969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:18.037197 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Nov 01 22:32:18.039969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:18.037266 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Nov 01 22:32:18.775105 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:32:18.775029 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:32:19.038886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:19.038312 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Nov 01 22:32:19.038886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:19.038385 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Nov 01 22:32:20.037877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:20.037036 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Nov 01 22:32:20.037877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:20.037102 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Nov 01 22:32:20.909918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:20.909878 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" event={"ID":"b1939254-2eef-4572-a8e1-6d075ab39eb9","Type":"ContainerStarted","Data":"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762"} Nov 01 22:32:20.910107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:20.909955 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:21.036879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:21.036341 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Nov 01 22:32:21.036879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:21.036406 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Nov 01 22:32:22.039335 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:22.039305 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:32:22.039739 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:22.039606 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:32:22.040395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:22.040378 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 22:32:22.062613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:22.062561 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" podStartSLOduration=3.171494071 podStartE2EDuration="7.062548307s" totalImagesPullingTime="3.891054236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:20.934753463 +0000 UTC m=+510.710740298" watchObservedRunningTime="2026-11-01 22:32:22.062548307 +0000 UTC m=+511.838535170" Nov 01 22:32:26.921845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:26.921815 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 22:32:38.847357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:38.847317 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fsf74"] Nov 01 22:32:38.852354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:38.852334 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:38.860534 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:38.859932 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fsf74"] Nov 01 22:32:38.957243 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:38.957202 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm49s\" (UniqueName: \"kubernetes.io/projected/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-kube-api-access-xm49s\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:38.957428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:38.957266 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-cert\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.057903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.057844 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm49s\" (UniqueName: \"kubernetes.io/projected/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-kube-api-access-xm49s\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.058095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.057916 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-cert\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.060527 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.060498 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-cert\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.066693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.066667 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm49s\" (UniqueName: \"kubernetes.io/projected/7c05e5e7-e375-4b2f-93ef-3ba630065e0c-kube-api-access-xm49s\") pod \"llmisvc-controller-manager-6466c554d8-fsf74\" (UID: \"7c05e5e7-e375-4b2f-93ef-3ba630065e0c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.166710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.166680 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:39.441051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:39.441027 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fsf74"] Nov 01 22:32:39.442780 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:32:39.442751 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c05e5e7_e375_4b2f_93ef_3ba630065e0c.slice/crio-dd89a074d4d5df457497171c32a7b7c798ff22695919dcf394f423e0c1461729 WatchSource:0}: Error finding container dd89a074d4d5df457497171c32a7b7c798ff22695919dcf394f423e0c1461729: Status 404 returned error can't find the container with id dd89a074d4d5df457497171c32a7b7c798ff22695919dcf394f423e0c1461729 Nov 01 22:32:40.010522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:40.010483 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" event={"ID":"7c05e5e7-e375-4b2f-93ef-3ba630065e0c","Type":"ContainerStarted","Data":"9750b390dde1dd952f5c166a1c0a6209f95d8ff363be435e31f3a5530bdbda4f"} Nov 01 22:32:40.011007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:40.010548 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" event={"ID":"7c05e5e7-e375-4b2f-93ef-3ba630065e0c","Type":"ContainerStarted","Data":"dd89a074d4d5df457497171c32a7b7c798ff22695919dcf394f423e0c1461729"} Nov 01 22:32:40.011007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:40.010578 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:32:40.031588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:32:40.031539 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" podStartSLOduration=1.685537832 podStartE2EDuration="2.031524526s" totalImagesPullingTime="345.986694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:40.029915464 +0000 UTC m=+529.805902294" watchObservedRunningTime="2026-11-01 22:32:40.031524526 +0000 UTC m=+529.807511355" Nov 01 22:32:48.773787 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:32:48.773750 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:33:11.017519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.017446 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fsf74" Nov 01 22:33:11.061567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.061532 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:33:11.061766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.061743 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" podUID="060f0e1b-c856-4227-b784-00e8e099f920" containerName="manager" containerID="cri-o://3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329" gracePeriod=30 Nov 01 22:33:11.318530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.318507 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:33:11.338998 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.338971 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\" (UniqueName: \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\") pod \"060f0e1b-c856-4227-b784-00e8e099f920\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " Nov 01 22:33:11.339120 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.339004 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\" (UniqueName: \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\") pod \"060f0e1b-c856-4227-b784-00e8e099f920\" (UID: \"060f0e1b-c856-4227-b784-00e8e099f920\") " Nov 01 22:33:11.341404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.341375 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert" pod "060f0e1b-c856-4227-b784-00e8e099f920" (UID: "060f0e1b-c856-4227-b784-00e8e099f920"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:11.341711 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.341683 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq" pod "060f0e1b-c856-4227-b784-00e8e099f920" (UID: "060f0e1b-c856-4227-b784-00e8e099f920"). InnerVolumeSpecName "kube-api-access-n6wqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:11.440336 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.440304 5778 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/060f0e1b-c856-4227-b784-00e8e099f920-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.440336 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:11.440332 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wqq\" (UniqueName: \"kubernetes.io/projected/060f0e1b-c856-4227-b784-00e8e099f920-kube-api-access-n6wqq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:12.180226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.180192 5778 generic.go:356] "Generic (PLEG): container finished" podID="060f0e1b-c856-4227-b784-00e8e099f920" containerID="3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329" exitCode=0 Nov 01 22:33:12.180666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.180240 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" Nov 01 22:33:12.180666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.180274 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" event={"ID":"060f0e1b-c856-4227-b784-00e8e099f920","Type":"ContainerDied","Data":"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329"} Nov 01 22:33:12.180666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.180333 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6" event={"ID":"060f0e1b-c856-4227-b784-00e8e099f920","Type":"ContainerDied","Data":"9eb4e966c1a4f232e8042369909abce8bd1776fc0b97c6d616a59427ad3deb4f"} Nov 01 22:33:12.180666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.180363 5778 scope.go:122] "RemoveContainer" containerID="3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329" Nov 01 22:33:12.190105 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.190086 5778 scope.go:122] "RemoveContainer" containerID="3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329" Nov 01 22:33:12.190366 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:33:12.190347 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329\": container with ID starting with 3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329 not found: ID does not exist" containerID="3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329" Nov 01 22:33:12.190438 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.190373 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329"} err="failed to get container status \"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329\": rpc error: code = NotFound desc = could not find container \"3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329\": container with ID starting with 3c4d0b0952ed80a6bc6f833cd45dc1efaff1f3600e87fdfef9a962d07c3e8329 not found: ID does not exist" Nov 01 22:33:12.201496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.201470 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:33:12.204589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.204570 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nrqf6"] Nov 01 22:33:12.372642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:12.372606 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060f0e1b-c856-4227-b784-00e8e099f920" path="/var/lib/kubelet/pods/060f0e1b-c856-4227-b784-00e8e099f920/volumes" Nov 01 22:33:18.772911 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:33:18.772864 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:33:48.771768 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:33:48.771730 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:33:50.434617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:50.434595 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:33:50.435013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:33:50.434762 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:34:16.041271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.041235 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:34:16.041655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.041641 5778 cpu_manager.go:414] "containerMap: removing container" podUID="060f0e1b-c856-4227-b784-00e8e099f920" containerName="manager" Nov 01 22:34:16.041709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.041657 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060f0e1b-c856-4227-b784-00e8e099f920" containerName="manager" Nov 01 22:34:16.041771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.041762 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="060f0e1b-c856-4227-b784-00e8e099f920" containerName="manager" Nov 01 22:34:16.045239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.045212 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.058219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.058187 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:34:16.145347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145315 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145365 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145382 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145435 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145466 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145609 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145634 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145674 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.145710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.145698 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.246793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246758 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246821 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246845 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246897 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246920 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246959 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.246983 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247038 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247147 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247282 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247282 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247419 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247497 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.247838 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.247812 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.249552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.249523 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.249898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.249879 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.255621 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.255590 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.255890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.255844 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-48tss\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.364221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.364130 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:16.649937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.649909 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:34:16.651775 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:34:16.651750 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f46718b_8c9e_4d01_9486_e1a76d295976.slice/crio-b4c59b05a7c9016f7d6ecdd7a3af571f0f0d01c8a2fb3488703adf26a251a714 WatchSource:0}: Error finding container b4c59b05a7c9016f7d6ecdd7a3af571f0f0d01c8a2fb3488703adf26a251a714: Status 404 returned error can't find the container with id b4c59b05a7c9016f7d6ecdd7a3af571f0f0d01c8a2fb3488703adf26a251a714 Nov 01 22:34:16.654055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.654022 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:16.654141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.654088 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:16.654141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:16.654116 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:17.562352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:17.562318 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" event={"ID":"3f46718b-8c9e-4d01-9486-e1a76d295976","Type":"ContainerStarted","Data":"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a"} Nov 01 22:34:17.562886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:17.562372 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" event={"ID":"3f46718b-8c9e-4d01-9486-e1a76d295976","Type":"ContainerStarted","Data":"b4c59b05a7c9016f7d6ecdd7a3af571f0f0d01c8a2fb3488703adf26a251a714"} Nov 01 22:34:17.585334 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:17.585271 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podStartSLOduration=1.585256459 podStartE2EDuration="1.585256459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:17.582556831 +0000 UTC m=+627.358543659" watchObservedRunningTime="2026-11-01 22:34:17.585256459 +0000 UTC m=+627.361243287" Nov 01 22:34:18.365225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:18.365180 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:18.366962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:18.366938 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.367089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:18.366999 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:18.773616 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:34:18.773575 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:34:19.365473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:19.365436 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.365661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:19.365492 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:20.364813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:20.364777 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.365328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:20.364834 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:21.365293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:21.365256 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.365706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:21.365315 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:22.364973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:22.364943 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:22.365151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:22.364993 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:23.364992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:23.364956 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:34:23.365385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:23.365014 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:34:24.849546 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:24.849516 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:24.850060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:24.849970 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:24.850976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:24.850958 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:34:31.053874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.053761 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj"] Nov 01 22:34:31.058413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.058367 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.068679 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.068656 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj"] Nov 01 22:34:31.181792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.181754 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drx7\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-kube-api-access-9drx7\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.181815 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.181965 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.181994 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.182018 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.182094 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.182137 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.182180 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.182383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.182233 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283708 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283674 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drx7\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-kube-api-access-9drx7\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283728 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283759 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283773 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283793 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283814 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283828 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283864 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.283931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.283896 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.284373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.284313 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.284373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.284347 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.284446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.284413 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.284479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.284449 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.284714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.284695 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.286373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.286354 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.286711 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.286691 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.292931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.292910 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.293011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.292951 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drx7\" (UniqueName: \"kubernetes.io/projected/e6e8ec16-0274-4d53-8fcf-bc8df04dbed8-kube-api-access-9drx7\") pod \"maas-default-gateway-openshift-default-6886b96f9-bftkj\" (UID: \"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.373923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.373814 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:31.663880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.658511 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj"] Nov 01 22:34:31.667202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.667161 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:31.667341 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.667251 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:31.667341 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:31.667311 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:32.644998 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:32.644965 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" event={"ID":"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8","Type":"ContainerStarted","Data":"ad5d05b8fdb828d21f8d86d42867592837a6049bc73a255a8c6cfacada7e47d3"} Nov 01 22:34:32.645490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:32.645032 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" event={"ID":"e6e8ec16-0274-4d53-8fcf-bc8df04dbed8","Type":"ContainerStarted","Data":"53f3c0e35885c5272e24bc4a16502fede98c07ad24bcbf6524a8dab31b9f4487"} Nov 01 22:34:32.670585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:32.670539 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podStartSLOduration=1.6705259730000002 podStartE2EDuration="1.670525973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:32.669924589 +0000 UTC m=+642.445911417" watchObservedRunningTime="2026-11-01 22:34:32.670525973 +0000 UTC m=+642.446512901" Nov 01 22:34:33.374569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:33.374532 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:33.375734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:33.375708 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:33.375894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:33.375756 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:34.374301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:34.374271 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:34.374705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:34.374322 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:35.374772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:35.374732 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:35.375215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:35.374797 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:36.374463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:36.374430 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:36.374626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:36.374482 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:37.374538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:37.374498 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:37.375076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:37.374560 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:38.374792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:38.374752 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:34:38.375204 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:38.374815 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 22:34:39.794830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:39.794798 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:39.795293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:39.795250 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:39.796002 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:34:39.795983 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" Nov 01 22:34:48.772160 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:34:48.772116 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:35:00.896918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.896880 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 22:35:00.901048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.901026 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:00.908223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.908065 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-927b68-dockercfg-dgjpw\"" Nov 01 22:35:00.908475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.908457 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-927b68\"" Nov 01 22:35:00.917655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.917631 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 22:35:00.972037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.972003 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 22:35:00.975439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.975404 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:00.975573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.975468 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkv84\" (UniqueName: \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:00.976104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.976086 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:00.993363 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:00.993337 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 22:35:01.076906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.076835 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:01.076906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.076908 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf6l\" (UniqueName: \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.077160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.076943 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkv84\" (UniqueName: \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:01.077160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.076959 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.077764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.077737 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:01.090994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.090958 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkv84\" (UniqueName: \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\") pod \"payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:01.177617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.177578 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf6l\" (UniqueName: \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.177800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.177632 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.178416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.178396 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.187570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.187468 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf6l\" (UniqueName: \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\") pod \"payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.214268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.214241 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:01.290250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.290129 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:01.428991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.428912 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 22:35:01.433226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.433203 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.437638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.437616 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e26733-serving-cert\"" Nov 01 22:35:01.447202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.447177 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 22:35:01.480622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.480582 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.480824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.480652 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.480943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.480865 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.480943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.480914 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.495896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.495869 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 22:35:01.500252 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:01.500227 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a753a9_2199_49ce_a1e2_3292ec66f1ab.slice/crio-9e200e6d1f1b4c5db3f2c1b6fc52cec2185fcd43ee13bbb76e730ac476d18cc6 WatchSource:0}: Error finding container 9e200e6d1f1b4c5db3f2c1b6fc52cec2185fcd43ee13bbb76e730ac476d18cc6: Status 404 returned error can't find the container with id 9e200e6d1f1b4c5db3f2c1b6fc52cec2185fcd43ee13bbb76e730ac476d18cc6 Nov 01 22:35:01.502087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.502073 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:35:01.582433 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.582395 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.582634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.582451 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.582634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.582511 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.582634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.582550 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.583369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.583338 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.585078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.585047 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.585078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.585065 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.590454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.590430 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 22:35:01.592686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.592597 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\") pod \"maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.594968 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:01.594927 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda429d996_5732_4237_878a_cb5c498a7570.slice/crio-c50244f509021f92dbb28716160427d08ce0e2e57dec393f5d400c78531baee4 WatchSource:0}: Error finding container c50244f509021f92dbb28716160427d08ce0e2e57dec393f5d400c78531baee4: Status 404 returned error can't find the container with id c50244f509021f92dbb28716160427d08ce0e2e57dec393f5d400c78531baee4 Nov 01 22:35:01.747532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.747440 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:01.805479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.805178 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" event={"ID":"58a753a9-2199-49ce-a1e2-3292ec66f1ab","Type":"ContainerStarted","Data":"9e200e6d1f1b4c5db3f2c1b6fc52cec2185fcd43ee13bbb76e730ac476d18cc6"} Nov 01 22:35:01.806959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:01.806697 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" event={"ID":"a429d996-5732-4237-878a-cb5c498a7570","Type":"ContainerStarted","Data":"c50244f509021f92dbb28716160427d08ce0e2e57dec393f5d400c78531baee4"} Nov 01 22:35:02.044178 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.044144 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 22:35:02.045115 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:02.045077 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efec5dc_2357_4a61_9fe5_3e73b516be01.slice/crio-aa9456d18ba383e5169dcb7ed30dde0ed43f5ca45b1b4a3633912af01af9a894 WatchSource:0}: Error finding container aa9456d18ba383e5169dcb7ed30dde0ed43f5ca45b1b4a3633912af01af9a894: Status 404 returned error can't find the container with id aa9456d18ba383e5169dcb7ed30dde0ed43f5ca45b1b4a3633912af01af9a894 Nov 01 22:35:02.817551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.817465 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" event={"ID":"5efec5dc-2357-4a61-9fe5-3e73b516be01","Type":"ContainerStarted","Data":"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c"} Nov 01 22:35:02.817551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.817527 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" event={"ID":"5efec5dc-2357-4a61-9fe5-3e73b516be01","Type":"ContainerStarted","Data":"aa9456d18ba383e5169dcb7ed30dde0ed43f5ca45b1b4a3633912af01af9a894"} Nov 01 22:35:02.818369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.818314 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:02.836535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.836503 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 22:35:02.841051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.841032 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:02.850977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.850760 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f338e4-dockercfg-vf42d\"" Nov 01 22:35:02.850977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.850885 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f338e4\"" Nov 01 22:35:02.856926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.856874 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 22:35:02.859573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.859522 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" podStartSLOduration=1.429218387 podStartE2EDuration="1.859505182s" totalImagesPullingTime="430.286795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.85851543 +0000 UTC m=+672.634502297" watchObservedRunningTime="2026-11-01 22:35:02.859505182 +0000 UTC m=+672.635492014" Nov 01 22:35:02.905062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.905017 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84ks7\" (UniqueName: \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:02.905245 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:02.905087 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.006901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.006843 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84ks7\" (UniqueName: \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.007103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.006930 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.007895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.007844 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.027911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.027760 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84ks7\" (UniqueName: \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\") pod \"payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.164422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.164124 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:03.505554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.505344 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 22:35:03.508646 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:03.508613 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce72afa2_7f2e_43f2_98fa_6d38973ada6b.slice/crio-5f4d8ec9fdc03748b5dca7f55157c77f9da8df6df89834273374ec8cce37bd1f WatchSource:0}: Error finding container 5f4d8ec9fdc03748b5dca7f55157c77f9da8df6df89834273374ec8cce37bd1f: Status 404 returned error can't find the container with id 5f4d8ec9fdc03748b5dca7f55157c77f9da8df6df89834273374ec8cce37bd1f Nov 01 22:35:03.829234 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:03.829143 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" event={"ID":"ce72afa2-7f2e-43f2-98fa-6d38973ada6b","Type":"ContainerStarted","Data":"5f4d8ec9fdc03748b5dca7f55157c77f9da8df6df89834273374ec8cce37bd1f"} Nov 01 22:35:04.617786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.617754 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 22:35:04.642924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.642139 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.643085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.642947 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 22:35:04.649298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.649270 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-927b68-serving-cert\"" Nov 01 22:35:04.736372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.726997 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ntf\" (UniqueName: \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.736372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.727060 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.736372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.727151 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.736372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.727231 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.837590 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.837517 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.837726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.837591 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ntf\" (UniqueName: \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.837726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.837628 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.837726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.837704 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.838281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.838249 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.841347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.841215 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.841347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.841258 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" event={"ID":"58a753a9-2199-49ce-a1e2-3292ec66f1ab","Type":"ContainerStarted","Data":"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962"} Nov 01 22:35:04.841347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.841316 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:04.842498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.842477 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.848665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.848616 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ntf\" (UniqueName: \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\") pod \"maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:04.865904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.865378 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" podStartSLOduration=1.787943818 podStartE2EDuration="4.865314414s" totalImagesPullingTime="3.077370596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:04.864606003 +0000 UTC m=+674.640592848" watchObservedRunningTime="2026-11-01 22:35:04.865314414 +0000 UTC m=+674.641301243" Nov 01 22:35:04.969254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:04.969220 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:05.261219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.261194 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 22:35:05.264266 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:05.264236 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe315b0_6760_49d9_b333_5531f0c66cad.slice/crio-8ca10637696bf8d4fb03f9fa9415a8c8478af9df76027ee14b92dc9f6350bc6f WatchSource:0}: Error finding container 8ca10637696bf8d4fb03f9fa9415a8c8478af9df76027ee14b92dc9f6350bc6f: Status 404 returned error can't find the container with id 8ca10637696bf8d4fb03f9fa9415a8c8478af9df76027ee14b92dc9f6350bc6f Nov 01 22:35:05.847974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.847917 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" event={"ID":"8fe315b0-6760-49d9-b333-5531f0c66cad","Type":"ContainerStarted","Data":"05f46f491c5e9304447938fcd3039ec96757940d1e4cd51356ea19673cba99bf"} Nov 01 22:35:05.848424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.847987 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" event={"ID":"8fe315b0-6760-49d9-b333-5531f0c66cad","Type":"ContainerStarted","Data":"8ca10637696bf8d4fb03f9fa9415a8c8478af9df76027ee14b92dc9f6350bc6f"} Nov 01 22:35:05.848424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.848125 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:05.849905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.849845 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" event={"ID":"ce72afa2-7f2e-43f2-98fa-6d38973ada6b","Type":"ContainerStarted","Data":"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49"} Nov 01 22:35:05.850030 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.849924 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:05.851673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.851646 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" event={"ID":"a429d996-5732-4237-878a-cb5c498a7570","Type":"ContainerStarted","Data":"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50"} Nov 01 22:35:05.851818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.851769 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:05.869489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.869438 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" podStartSLOduration=1.459119444 podStartE2EDuration="1.869425896s" totalImagesPullingTime="410.306452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:05.867260886 +0000 UTC m=+675.643247726" watchObservedRunningTime="2026-11-01 22:35:05.869425896 +0000 UTC m=+675.645412724" Nov 01 22:35:05.886167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.886123 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" podStartSLOduration=2.702892451 podStartE2EDuration="5.886111733s" totalImagesPullingTime="3.183219282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:05.883819821 +0000 UTC m=+675.659806662" watchObservedRunningTime="2026-11-01 22:35:05.886111733 +0000 UTC m=+675.662098575" Nov 01 22:35:05.903711 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:05.903656 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" podStartSLOduration=1.995191111 podStartE2EDuration="3.903638117s" totalImagesPullingTime="1.908447006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:05.900648871 +0000 UTC m=+675.676635712" watchObservedRunningTime="2026-11-01 22:35:05.903638117 +0000 UTC m=+675.679624946" Nov 01 22:35:06.529539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.527390 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 22:35:06.532394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.532354 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.545833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.544823 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 22:35:06.545966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.545950 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-6ac182-serving-cert\"" Nov 01 22:35:06.656492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.656449 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.656697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.656624 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5j2\" (UniqueName: \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.656697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.656672 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.656787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.656733 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.757367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.757329 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.757584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.757385 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.757584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.757476 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5j2\" (UniqueName: \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.757584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.757536 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.758151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.758117 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.760174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.760149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.760319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.760305 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.765953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.765925 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5j2\" (UniqueName: \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\") pod \"maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:06.853719 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:06.853637 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:07.144893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:07.144868 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 22:35:07.146911 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:07.146884 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c149a45_0908_430e_9f23_c3dabb1b9ea7.slice/crio-ca48b8de6fbff572f14ed204fe77c63fbe16086a2d96cfd7aab450bfce01235e WatchSource:0}: Error finding container ca48b8de6fbff572f14ed204fe77c63fbe16086a2d96cfd7aab450bfce01235e: Status 404 returned error can't find the container with id ca48b8de6fbff572f14ed204fe77c63fbe16086a2d96cfd7aab450bfce01235e Nov 01 22:35:07.865476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:07.865372 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" event={"ID":"2c149a45-0908-430e-9f23-c3dabb1b9ea7","Type":"ContainerStarted","Data":"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f"} Nov 01 22:35:07.865476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:07.865437 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" event={"ID":"2c149a45-0908-430e-9f23-c3dabb1b9ea7","Type":"ContainerStarted","Data":"ca48b8de6fbff572f14ed204fe77c63fbe16086a2d96cfd7aab450bfce01235e"} Nov 01 22:35:07.866051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:07.865556 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:07.887068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:07.887018 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" podStartSLOduration=1.49638359 podStartE2EDuration="1.887001438s" totalImagesPullingTime="390.617848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:07.885240236 +0000 UTC m=+677.661227066" watchObservedRunningTime="2026-11-01 22:35:07.887001438 +0000 UTC m=+677.662988280" Nov 01 22:35:09.848337 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:09.848302 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 22:35:11.863862 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:11.863817 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 22:35:13.877692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:13.877613 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 22:35:15.855837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:15.855806 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 22:35:16.861613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:16.861578 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 22:35:16.862879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:16.862819 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 22:35:18.794572 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:35:18.794530 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:35:24.839664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.839628 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 22:35:24.847628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.847606 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.859802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.859777 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 22:35:24.863894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.863871 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"default-dockercfg-pb82b\"" Nov 01 22:35:24.864031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.863996 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"simulator-w3-6ac182-kserve-self-signed-certs\"" Nov 01 22:35:24.941285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.941252 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"openshift-service-ca.crt\"" Nov 01 22:35:24.941285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.941265 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"kube-root-ca.crt\"" Nov 01 22:35:24.964681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964647 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.964875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964779 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.964875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964823 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.964875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964860 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.965057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964892 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8j7\" (UniqueName: \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:24.965057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:24.964978 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066280 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066382 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066422 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066449 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066490 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8j7\" (UniqueName: \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066516 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066767 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.066892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.066874 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.067134 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.067116 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.069620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.069596 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.069749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.069728 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.080051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.080007 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8j7\" (UniqueName: \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\") pod \"simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.161709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.161680 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:25.451923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.451897 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 22:35:25.454047 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:25.454009 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b50e76_eec9_41fa_8a9d_c708e67ac7a2.slice/crio-e7d5feb741af10a3825de23b9fa83f1083034faef1a07384a056fb03a7d127fd WatchSource:0}: Error finding container e7d5feb741af10a3825de23b9fa83f1083034faef1a07384a056fb03a7d127fd: Status 404 returned error can't find the container with id e7d5feb741af10a3825de23b9fa83f1083034faef1a07384a056fb03a7d127fd Nov 01 22:35:25.991215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:25.991181 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" event={"ID":"81b50e76-eec9-41fa-8a9d-c708e67ac7a2","Type":"ContainerStarted","Data":"e7d5feb741af10a3825de23b9fa83f1083034faef1a07384a056fb03a7d127fd"} Nov 01 22:35:28.006146 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:28.006114 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" event={"ID":"81b50e76-eec9-41fa-8a9d-c708e67ac7a2","Type":"ContainerStarted","Data":"445481322dd104dc5497902b4b20f6d4f93553de1a783ef35ccc14e8c5ce5a8b"} Nov 01 22:35:28.034904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:28.034833 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podStartSLOduration=2.357060332 podStartE2EDuration="4.03481792s" totalImagesPullingTime="1.677757588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:28.031934861 +0000 UTC m=+697.807921692" watchObservedRunningTime="2026-11-01 22:35:28.03481792 +0000 UTC m=+697.810804748" Nov 01 22:35:28.162600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:28.162566 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:35.162067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:35.162019 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:35.180756 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:35.180723 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:36.072328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:36.072275 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:35:38.641398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.641365 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 22:35:38.647051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.647028 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.656532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.656507 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 22:35:38.656782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.656753 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"kube-root-ca.crt\"" Nov 01 22:35:38.657260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.657241 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"openshift-service-ca.crt\"" Nov 01 22:35:38.658299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.658223 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"default-dockercfg-ghrq6\"" Nov 01 22:35:38.658398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.658377 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"premium-w4-2f3705-kserve-self-signed-certs\"" Nov 01 22:35:38.693640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.693609 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.693767 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.693679 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.693814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.693770 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.693883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.693810 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.693990 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.693972 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.694037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.694020 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhknb\" (UniqueName: \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795443 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795402 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795500 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795540 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795566 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795693 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795749 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhknb\" (UniqueName: \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.795970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.795896 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.796416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.796394 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.796583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.796560 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.799068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.799043 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.800271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.800245 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.805462 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.805428 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhknb\" (UniqueName: \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\") pod \"premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:38.963563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:38.963526 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:39.246579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:39.246548 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 22:35:39.247808 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:39.247781 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e6d3ca2_80af_4215_b87a_22970a3409d7.slice/crio-572991aa19124cc3445c4fd44523a2f5d8d9f2ddfda2f948a029ca3344b1da8a WatchSource:0}: Error finding container 572991aa19124cc3445c4fd44523a2f5d8d9f2ddfda2f948a029ca3344b1da8a: Status 404 returned error can't find the container with id 572991aa19124cc3445c4fd44523a2f5d8d9f2ddfda2f948a029ca3344b1da8a Nov 01 22:35:40.083884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:40.083824 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" event={"ID":"3e6d3ca2-80af-4215-b87a-22970a3409d7","Type":"ContainerStarted","Data":"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b"} Nov 01 22:35:40.084361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:40.083894 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" event={"ID":"3e6d3ca2-80af-4215-b87a-22970a3409d7","Type":"ContainerStarted","Data":"572991aa19124cc3445c4fd44523a2f5d8d9f2ddfda2f948a029ca3344b1da8a"} Nov 01 22:35:40.112037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:40.111984 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podStartSLOduration=2.111972324 podStartE2EDuration="2.111972324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:40.108430417 +0000 UTC m=+709.884417245" watchObservedRunningTime="2026-11-01 22:35:40.111972324 +0000 UTC m=+709.887959152" Nov 01 22:35:40.964141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:40.964110 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:48.772687 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:35:48.772652 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:35:48.964257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:48.964216 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:48.977420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:48.977389 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:49.155175 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:49.155146 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:35:51.527580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.527545 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 22:35:51.532801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.532781 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.551283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.551258 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"kube-root-ca.crt\"" Nov 01 22:35:51.551458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.551259 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"premium-w2-9ad303-kserve-self-signed-certs\"" Nov 01 22:35:51.551458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.551348 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"openshift-service-ca.crt\"" Nov 01 22:35:51.551629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.551612 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 22:35:51.552473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.552459 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"default-dockercfg-p5mnk\"" Nov 01 22:35:51.625765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.625728 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.625765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.625778 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.626043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.625976 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqxz\" (UniqueName: \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.626043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.626036 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.626163 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.626138 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.626217 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.626176 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.726830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726789 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.726830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726837 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726901 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqxz\" (UniqueName: \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726933 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726954 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.726975 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.727557 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.727584 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.727948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.727724 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.729646 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.729619 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.730015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.729995 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.736320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.736291 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqxz\" (UniqueName: \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\") pod \"premium-w2-9ad303-kserve-d8668d79d-lb8d9\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:51.846745 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:51.846660 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:52.131839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.131817 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 22:35:52.134185 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:52.134148 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc478da01_6384_454f_b3ce_e521652a51ff.slice/crio-f544e27583e8a8905d7708df3c6f01008460a3bb85469b168898576fc7e6bad5 WatchSource:0}: Error finding container f544e27583e8a8905d7708df3c6f01008460a3bb85469b168898576fc7e6bad5: Status 404 returned error can't find the container with id f544e27583e8a8905d7708df3c6f01008460a3bb85469b168898576fc7e6bad5 Nov 01 22:35:52.157636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.157598 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" event={"ID":"c478da01-6384-454f-b3ce-e521652a51ff","Type":"ContainerStarted","Data":"f544e27583e8a8905d7708df3c6f01008460a3bb85469b168898576fc7e6bad5"} Nov 01 22:35:52.761780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.761746 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:35:52.765624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.765598 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.771547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.771525 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2f3705-openshift-default-dockercfg-29c4c\"" Nov 01 22:35:52.797671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.797489 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:35:52.840301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840272 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840305 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840340 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840389 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840470 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840493 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840694 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840531 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840694 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840666 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwznl\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.840779 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.840722 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.941970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.941931 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.941987 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942039 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942094 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwznl\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942137 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942477 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942454 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.942538 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942483 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942625 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942702 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.942737 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.943089 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.943145 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.943216 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.943686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.943656 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.944815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.944795 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.945004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.944984 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.951321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.951293 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:52.951610 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:52.951592 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwznl\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:53.085268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.085192 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:53.171530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.171493 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" event={"ID":"c478da01-6384-454f-b3ce-e521652a51ff","Type":"ContainerStarted","Data":"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73"} Nov 01 22:35:53.188354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.188309 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podStartSLOduration=2.188294784 podStartE2EDuration="2.188294784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:53.187714257 +0000 UTC m=+722.963701087" watchObservedRunningTime="2026-11-01 22:35:53.188294784 +0000 UTC m=+722.964281613" Nov 01 22:35:53.388411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.388377 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:35:53.391183 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:35:53.391152 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3973874_a07c_43dc_ab4f_f81b3ed0c5dd.slice/crio-0c1c11ce44eb01c5a1aa7db7578a16d64a5b8dc0f2cbf915b94b7ebd65c0af87 WatchSource:0}: Error finding container 0c1c11ce44eb01c5a1aa7db7578a16d64a5b8dc0f2cbf915b94b7ebd65c0af87: Status 404 returned error can't find the container with id 0c1c11ce44eb01c5a1aa7db7578a16d64a5b8dc0f2cbf915b94b7ebd65c0af87 Nov 01 22:35:53.395155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.395120 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:35:53.395271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.395193 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:35:53.395271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.395223 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:35:53.847732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:53.847696 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:35:54.186872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:54.186339 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" event={"ID":"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd","Type":"ContainerStarted","Data":"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f"} Nov 01 22:35:54.186872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:54.186390 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" event={"ID":"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd","Type":"ContainerStarted","Data":"0c1c11ce44eb01c5a1aa7db7578a16d64a5b8dc0f2cbf915b94b7ebd65c0af87"} Nov 01 22:35:54.206415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:54.206360 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podStartSLOduration=2.206341493 podStartE2EDuration="2.206341493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:54.20613563 +0000 UTC m=+723.982122472" watchObservedRunningTime="2026-11-01 22:35:54.206341493 +0000 UTC m=+723.982328321" Nov 01 22:35:55.086107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:55.086073 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:35:55.087578 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:55.087542 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:35:55.087732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:55.087597 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 22:35:56.085983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:56.085939 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:35:56.086189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:56.086036 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 22:35:57.086043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:57.086001 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:35:57.086249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:57.086091 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 22:35:58.086249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:58.086211 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:35:58.086672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:58.086285 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 22:35:59.085989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:59.085901 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:35:59.085989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:35:59.085957 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 22:36:00.380190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:00.380161 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:36:00.380688 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:00.380625 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:36:00.381575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:00.381546 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:36:01.847366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:01.847329 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:36:01.860029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:01.860001 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:36:02.264048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:02.264016 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:36:07.643890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.643839 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 22:36:07.648112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.648094 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.656199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.656174 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e26733-openshift-default-dockercfg-794k8\"" Nov 01 22:36:07.662585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.662562 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 22:36:07.822911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.822879 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.822942 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823070 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823159 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823266 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823250 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823289 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823315 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823339 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lncz2\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.823419 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.823370 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924277 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924363 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924409 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924448 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924482 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924569 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924602 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924629 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924651 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncz2\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.924777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924706 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.925083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.924999 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.925410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.925387 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.925601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.925565 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.925987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.925961 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.928087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.928065 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.928312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.928291 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.934505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.934482 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.934893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.934873 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lncz2\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:07.965929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:07.965034 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:08.273916 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:08.273486 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 22:36:08.285171 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:36:08.285143 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc37c643_dc48_4601_b18a_acd7262df3f3.slice/crio-d176122ba6fb76bae9c877b3c401bba2aeff80fb23feee4a41b189767c420c24 WatchSource:0}: Error finding container d176122ba6fb76bae9c877b3c401bba2aeff80fb23feee4a41b189767c420c24: Status 404 returned error can't find the container with id d176122ba6fb76bae9c877b3c401bba2aeff80fb23feee4a41b189767c420c24 Nov 01 22:36:08.289502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:08.289463 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:36:08.289682 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:08.289663 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:36:08.289839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:08.289820 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:36:09.300063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.300022 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" event={"ID":"cc37c643-dc48-4601-b18a-acd7262df3f3","Type":"ContainerStarted","Data":"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752"} Nov 01 22:36:09.300572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.300084 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" event={"ID":"cc37c643-dc48-4601-b18a-acd7262df3f3","Type":"ContainerStarted","Data":"d176122ba6fb76bae9c877b3c401bba2aeff80fb23feee4a41b189767c420c24"} Nov 01 22:36:09.345808 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.345735 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podStartSLOduration=2.345715329 podStartE2EDuration="2.345715329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:09.336365308 +0000 UTC m=+739.112352172" watchObservedRunningTime="2026-11-01 22:36:09.345715329 +0000 UTC m=+739.121702159" Nov 01 22:36:09.966668 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.966633 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:09.968194 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.968162 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.968441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:09.968404 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 01 22:36:10.967214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:10.967177 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.967743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:10.967253 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 01 22:36:11.967020 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:11.966977 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.967223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:11.967045 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 01 22:36:12.966956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:12.966916 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.967360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:12.966972 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 01 22:36:13.966507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:13.966466 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.966684 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:13.966539 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 01 22:36:14.970406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:14.970377 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:14.970882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:14.970870 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:14.971700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:36:14.971677 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 22:36:18.773934 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:36:18.773899 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:36:48.774336 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:36:48.774298 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:37:07.282041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.282006 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg"] Nov 01 22:37:07.285975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.285956 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.296302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.296281 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:07.297481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.297462 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:07.300827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.300807 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg"] Nov 01 22:37:07.357318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.357286 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.357487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.357335 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsdq\" (UniqueName: \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.357487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.357464 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.458641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.458607 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.458839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.458736 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.458839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.458794 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsdq\" (UniqueName: \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.459516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.459492 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.461333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.461315 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.472782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.472755 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsdq\" (UniqueName: \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\") pod \"maas-api-revoke-sub-be62e240-vj7cg\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.599723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.599628 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:07.887473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:07.887439 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg"] Nov 01 22:37:07.889125 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:07.889101 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29f1133_7b63_46a9_b3bd_4ff7044be449.slice/crio-e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de WatchSource:0}: Error finding container e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de: Status 404 returned error can't find the container with id e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de Nov 01 22:37:08.692291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:08.692255 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" event={"ID":"e29f1133-7b63-46a9-b3bd-4ff7044be449","Type":"ContainerStarted","Data":"e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de"} Nov 01 22:37:11.716594 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:11.716494 5778 generic.go:356] "Generic (PLEG): container finished" podID="e29f1133-7b63-46a9-b3bd-4ff7044be449" containerID="a52c7df1750b0dd02b0f7efc2195b667b4998bf86e4436fac662268193c0da1c" exitCode=0 Nov 01 22:37:11.717158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:11.716612 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" event={"ID":"e29f1133-7b63-46a9-b3bd-4ff7044be449","Type":"ContainerDied","Data":"a52c7df1750b0dd02b0f7efc2195b667b4998bf86e4436fac662268193c0da1c"} Nov 01 22:37:13.210061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.210016 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:13.275207 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.275172 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\" (UniqueName: \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\") pod \"e29f1133-7b63-46a9-b3bd-4ff7044be449\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " Nov 01 22:37:13.275933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.275904 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\") pod \"e29f1133-7b63-46a9-b3bd-4ff7044be449\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " Nov 01 22:37:13.276285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.276264 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca" pod "e29f1133-7b63-46a9-b3bd-4ff7044be449" (UID: "e29f1133-7b63-46a9-b3bd-4ff7044be449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:13.276409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.276336 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\") pod \"e29f1133-7b63-46a9-b3bd-4ff7044be449\" (UID: \"e29f1133-7b63-46a9-b3bd-4ff7044be449\") " Nov 01 22:37:13.280929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.280905 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.414500 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.414251 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq" pod "e29f1133-7b63-46a9-b3bd-4ff7044be449" (UID: "e29f1133-7b63-46a9-b3bd-4ff7044be449"). InnerVolumeSpecName "kube-api-access-djsdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:13.414969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.414943 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca" pod "e29f1133-7b63-46a9-b3bd-4ff7044be449" (UID: "e29f1133-7b63-46a9-b3bd-4ff7044be449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:13.486805 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.485697 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29f1133-7b63-46a9-b3bd-4ff7044be449-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.486805 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.485734 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsdq\" (UniqueName: \"kubernetes.io/projected/e29f1133-7b63-46a9-b3bd-4ff7044be449-kube-api-access-djsdq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.758845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.758755 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" event={"ID":"e29f1133-7b63-46a9-b3bd-4ff7044be449","Type":"ContainerDied","Data":"e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de"} Nov 01 22:37:13.763177 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.763155 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59b3797bde6b9a70cfe3a80125e443ec4df55eefd2336199fe6a73637a977de" Nov 01 22:37:13.763366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:13.763350 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg" Nov 01 22:37:15.800119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.800084 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 22:37:15.802584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.800743 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e29f1133-7b63-46a9-b3bd-4ff7044be449" containerName="revoke-keys" Nov 01 22:37:15.802584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.800762 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29f1133-7b63-46a9-b3bd-4ff7044be449" containerName="revoke-keys" Nov 01 22:37:15.802584 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.800957 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29f1133-7b63-46a9-b3bd-4ff7044be449" containerName="revoke-keys" Nov 01 22:37:15.804527 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.804508 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.811442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.811417 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"distinct-w1-f338e4-kserve-self-signed-certs\"" Nov 01 22:37:15.811614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.811455 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"kube-root-ca.crt\"" Nov 01 22:37:15.811614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.811490 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"openshift-service-ca.crt\"" Nov 01 22:37:15.811775 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.811548 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"default-dockercfg-td692\"" Nov 01 22:37:15.820398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.820376 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 22:37:15.921840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.921802 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.922050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.921864 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqqg\" (UniqueName: \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.922050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.922021 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.922158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.922086 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.922158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.922121 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:15.922255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:15.922161 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023612 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023664 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023687 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023708 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023732 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.023883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.023753 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqqg\" (UniqueName: \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.024234 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.024131 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.024297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.024280 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.024435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.024413 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.026681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.026654 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.027973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.027950 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.034523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.034497 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqqg\" (UniqueName: \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\") pod \"distinct-w1-f338e4-kserve-86f5dd5869-8xn72\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.121222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.121140 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:16.408252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.408229 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 22:37:16.410636 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:16.410599 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b1296e_ee30_4ccb_8f99_96a3552178c9.slice/crio-500fea8c9c534070afd9e619a2cfcdda1868edff13dc47635f3d456d5a4005fb WatchSource:0}: Error finding container 500fea8c9c534070afd9e619a2cfcdda1868edff13dc47635f3d456d5a4005fb: Status 404 returned error can't find the container with id 500fea8c9c534070afd9e619a2cfcdda1868edff13dc47635f3d456d5a4005fb Nov 01 22:37:16.778531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.778446 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" event={"ID":"96b1296e-ee30-4ccb-8f99-96a3552178c9","Type":"ContainerStarted","Data":"b9994d9ec8785b7dcba1d73a94336b556bf47cc7a346f8b5bb08a43f2dc97301"} Nov 01 22:37:16.778531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.778489 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" event={"ID":"96b1296e-ee30-4ccb-8f99-96a3552178c9","Type":"ContainerStarted","Data":"500fea8c9c534070afd9e619a2cfcdda1868edff13dc47635f3d456d5a4005fb"} Nov 01 22:37:16.797215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:16.797161 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podStartSLOduration=1.797148389 podStartE2EDuration="1.797148389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:16.796009868 +0000 UTC m=+806.571996682" watchObservedRunningTime="2026-11-01 22:37:16.797148389 +0000 UTC m=+806.573135218" Nov 01 22:37:17.122113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:17.122024 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:18.774470 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:37:18.774437 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:37:23.139974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.139943 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 22:37:23.144209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.144184 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.151592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.151570 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-927b68-openshift-default-dockercfg-rfvvh\"" Nov 01 22:37:23.160223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.160202 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 22:37:23.203594 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203570 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.203748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203622 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947q9\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.203748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203731 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.203886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203771 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.203886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203802 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.204013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203892 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.204013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.203940 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.204126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.204022 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.204126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.204065 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305240 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305286 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947q9\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305328 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305349 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305363 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305451 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305520 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305571 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305612 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305745 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.305816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.305793 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.306082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.306058 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.306173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.306086 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.306173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.306116 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.308124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.308094 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.308402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.308384 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.314639 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.314616 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.315279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.315243 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947q9\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.463567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.463536 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 22:37:23.468441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.468421 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.475267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.475246 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6ac182-openshift-default-dockercfg-mf2q8\"" Nov 01 22:37:23.484362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.484339 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 22:37:23.495035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.495016 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:23.509019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.508995 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509203 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509240 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509277 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4mg\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509323 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509349 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509453 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509544 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.509616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.509568 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.610844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.610806 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.610876 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.610911 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.610944 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4mg\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.610987 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.611016 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.611111 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.611182 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.611316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.611205 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.612769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.612437 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.612769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.612707 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.613023 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.612935 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.613023 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.612977 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.613294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.613249 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.614014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.613989 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.615111 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.615088 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.620777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.620748 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.621690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.621665 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4mg\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.661021 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.660974 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 22:37:23.666723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.666700 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.681129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.681093 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ad303-openshift-default-dockercfg-cbwmc\"" Nov 01 22:37:23.686173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.686148 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 22:37:23.712585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712551 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.712723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712588 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.712723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712703 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.712829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712750 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.712911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712892 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.712969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.712927 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.713079 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.713064 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qch9\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.713140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.713093 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.713197 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.713138 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.782617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.782542 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:23.796078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.796055 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 22:37:23.798085 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:23.798058 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a93e086_8ae2_44b5_998c_b8d8e4d90618.slice/crio-292a0437addef89a78ecb61f8c6623e4a083d469741b37ecff886c5708f6c035 WatchSource:0}: Error finding container 292a0437addef89a78ecb61f8c6623e4a083d469741b37ecff886c5708f6c035: Status 404 returned error can't find the container with id 292a0437addef89a78ecb61f8c6623e4a083d469741b37ecff886c5708f6c035 Nov 01 22:37:23.800507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.800468 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:23.800582 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.800552 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:23.800634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.800591 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:23.824346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824309 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824431 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824459 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824515 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824544 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824601 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824629 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824736 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qch9\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.824964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.824765 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.829015 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.829294 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.829521 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.830760 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.831075 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.832599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.831488 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.839616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.839595 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.841327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.841302 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" event={"ID":"0a93e086-8ae2-44b5-998c-b8d8e4d90618","Type":"ContainerStarted","Data":"292a0437addef89a78ecb61f8c6623e4a083d469741b37ecff886c5708f6c035"} Nov 01 22:37:23.850439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.850416 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qch9\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.851092 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.851071 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:23.881892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.876656 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:37:23.884148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.884130 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.897673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.897645 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929395 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929442 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929487 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929555 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrrw\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929597 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929648 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929674 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929704 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.930022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.929750 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:23.935924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.931866 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f338e4-openshift-default-dockercfg-ksfnw\"" Nov 01 22:37:23.998537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:23.993097 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:24.032790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.032350 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.032790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.032394 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.033204 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.033181 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.034056 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.033925 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.034235 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.034219 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrrw\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.034672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.034650 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.035182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.035166 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.035623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.035606 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.036181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.036163 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.036631 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.036614 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.036982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.035067 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.037298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.036548 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.037408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.036516 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.037507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.037254 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.044693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.044669 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrrw\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.048373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.048332 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.051218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.051173 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.070147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.070091 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.097020 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.096993 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 22:37:24.108680 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:24.106618 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb312c02_f1b1_4eb0_809a_3bf6bdef2776.slice/crio-211619961b86558d9bf7b04694b4eb7a8fc317dc80e24949f81c7d4fb6b23f79 WatchSource:0}: Error finding container 211619961b86558d9bf7b04694b4eb7a8fc317dc80e24949f81c7d4fb6b23f79: Status 404 returned error can't find the container with id 211619961b86558d9bf7b04694b4eb7a8fc317dc80e24949f81c7d4fb6b23f79 Nov 01 22:37:24.111600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.111556 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.111715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.111647 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.111715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.111685 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.208721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.208440 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:24.353033 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.352753 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 22:37:24.359980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.359936 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.360109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.360055 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.360109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.360104 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.514250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.514220 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:37:24.516434 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:24.516410 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d76b9e_cb48_4bf3_aab0_ed31dfd3ef6f.slice/crio-0c457ed692ab93e8fc31a19cb3d6350b72f9cad6dae1ea6672d916853da4ac89 WatchSource:0}: Error finding container 0c457ed692ab93e8fc31a19cb3d6350b72f9cad6dae1ea6672d916853da4ac89: Status 404 returned error can't find the container with id 0c457ed692ab93e8fc31a19cb3d6350b72f9cad6dae1ea6672d916853da4ac89 Nov 01 22:37:24.522876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.522822 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.523059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.523037 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.523184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.523166 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:24.897455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.896341 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" event={"ID":"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f","Type":"ContainerStarted","Data":"3f16f8da0dca129afcd8c1af52cc2b116c939128c26d560e5bc3af1d7d921eb6"} Nov 01 22:37:24.897455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.896408 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" event={"ID":"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f","Type":"ContainerStarted","Data":"0c457ed692ab93e8fc31a19cb3d6350b72f9cad6dae1ea6672d916853da4ac89"} Nov 01 22:37:24.901870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.899940 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" event={"ID":"db312c02-f1b1-4eb0-809a-3bf6bdef2776","Type":"ContainerStarted","Data":"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318"} Nov 01 22:37:24.901870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.900055 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" event={"ID":"db312c02-f1b1-4eb0-809a-3bf6bdef2776","Type":"ContainerStarted","Data":"211619961b86558d9bf7b04694b4eb7a8fc317dc80e24949f81c7d4fb6b23f79"} Nov 01 22:37:24.906327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.906294 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" event={"ID":"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285","Type":"ContainerStarted","Data":"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a"} Nov 01 22:37:24.906554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.906532 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" event={"ID":"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285","Type":"ContainerStarted","Data":"b2a68eb4c5bef1ef7173175d97a2743fbdcbb703e94bc9d3d42b5e7694a0a4f3"} Nov 01 22:37:24.910385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.910346 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" event={"ID":"0a93e086-8ae2-44b5-998c-b8d8e4d90618","Type":"ContainerStarted","Data":"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743"} Nov 01 22:37:24.919685 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.919114 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podStartSLOduration=1.919096318 podStartE2EDuration="1.919096318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:24.918273721 +0000 UTC m=+814.694260547" watchObservedRunningTime="2026-11-01 22:37:24.919096318 +0000 UTC m=+814.695083148" Nov 01 22:37:24.987261 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.987202 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podStartSLOduration=1.987184159 podStartE2EDuration="1.987184159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:24.986355518 +0000 UTC m=+814.762342360" watchObservedRunningTime="2026-11-01 22:37:24.987184159 +0000 UTC m=+814.763170988" Nov 01 22:37:24.998215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:24.998176 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:25.001142 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.001089 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podStartSLOduration=2.001069767 podStartE2EDuration="2.001069767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:24.955156917 +0000 UTC m=+814.731143747" watchObservedRunningTime="2026-11-01 22:37:25.001069767 +0000 UTC m=+814.777056597" Nov 01 22:37:25.015193 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.015149 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podStartSLOduration=2.015135633 podStartE2EDuration="2.015135633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:25.010462728 +0000 UTC m=+814.786449591" watchObservedRunningTime="2026-11-01 22:37:25.015135633 +0000 UTC m=+814.791122462" Nov 01 22:37:25.209938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.209902 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:25.210355 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.210002 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.210355 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.210051 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:25.499764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.499676 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:25.500144 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.500119 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.500330 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.500305 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:25.787932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.784793 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:25.787932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.784963 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.787932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:25.785012 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:26.003872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.003062 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.003872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.003118 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:26.131068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.129287 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:26.210318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.210279 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.210962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.210925 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:26.502280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.502243 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.502523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.502500 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:26.610875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.610199 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:26.785523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.785434 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.785802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.785760 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:27.003015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:26.999954 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.003015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.000038 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:27.222347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.222303 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.223000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.222963 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:27.502452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.502352 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.502733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.502703 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:27.787432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.787338 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.787691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:27.787662 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:28.002917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.002289 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.002917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.002356 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:28.105060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.104952 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.144878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.142944 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.222879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.221970 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.222879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.222039 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:28.504320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.504280 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.504615 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.504586 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:28.785339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.785221 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.785661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.785539 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:28.998318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.998283 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.998549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:28.998527 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:29.069197 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.069104 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:29.076826 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.076788 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.161895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.146000 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.161895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.146943 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:37:29.213360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.213325 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.213948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.213573 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:29.500285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.500240 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.508996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.508954 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:29.812721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.812629 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.815883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.812963 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:30.010875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:29.968365 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:37:30.013817 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.012365 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.014073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.014050 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:30.463877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.462992 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.463877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.463074 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:30.498946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.498907 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.499146 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.498976 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:30.818930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.812790 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.818930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.812876 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:30.998349 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.998315 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.998609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:30.998582 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:31.116068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.115951 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.214298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.214259 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.214638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.214598 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:31.253876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.252157 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.499269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.499229 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.638135 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.638084 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:31.784422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.784326 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.792876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.784647 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:31.953029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:31.952970 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.002370 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.002335 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.002683 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.002635 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:32.058104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.057991 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.132879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.126469 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.132879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.127238 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:32.219041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.219002 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.219831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.219786 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:32.300522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.300478 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.306515 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.306485 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:37:32.601908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.598433 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.601908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.598511 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:32.870547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.870440 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.870904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:32.870844 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:33.109880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.102599 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:37:33.136019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.135921 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:33.138911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.138876 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:33.223918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.219977 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:33.223918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.220048 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:33.318375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.318346 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:37:33.506406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.506371 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:33.506725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.506678 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:33.808978 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.807405 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:33.808978 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:33.807464 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:34.014020 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.007050 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:34.014020 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.007181 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:34.212401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.212363 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:34.213077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.213029 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:34.511707 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.511233 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:34.511707 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.511293 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:34.789934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.785438 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:37:34.789934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:34.785516 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 01 22:37:35.003879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.003390 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:35.003879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.003472 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:35.212790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.212292 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:35.212790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.212406 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:35.499457 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.499352 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:35.499800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:35.499756 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 01 22:37:36.006689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.006656 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:36.007013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.006982 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 22:37:36.254060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.254002 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 22:37:36.254665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.254632 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 22:37:36.359903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.359784 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:36.361015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.360998 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:36.365300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:36.365268 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 22:37:37.539015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:37.538975 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:37.539591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:37.539557 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:37.687745 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:37.686178 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:37.687745 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:37.687563 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:37.691581 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:37.691550 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 22:37:38.041062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.041021 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:38.041372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.041333 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:38.262914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.259957 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:38.262914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.260026 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:38.553725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.553412 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:38.555365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.554899 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:38.564125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.560032 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 22:37:38.789401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.789370 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:38.789985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.789967 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:38.791238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:38.791218 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:37:48.773890 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:37:48.773828 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:37:52.819405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.819370 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:37:52.827159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.827138 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.836385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.836350 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:37:52.980355 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980324 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980374 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980394 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980528 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980562 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980578 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26c7j\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980683 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980592 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980734 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:52.980800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:52.980789 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082102 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082166 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082223 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082252 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082309 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082353 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082369 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26c7j\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082386 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082448 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.082589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.082552 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.083030 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.083006 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.083385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.083014 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.083385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.083379 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.083527 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.083089 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.085241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.085198 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.085385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.085365 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.095604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.095580 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.096496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.096462 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26c7j\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.144938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.144910 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:53.441144 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.441108 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:37:53.441741 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:37:53.441656 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ec7b7a_a00c_4206_98b1_034896709f93.slice/crio-d0c8d72113209c3ea0abb4d22ce2ceb9c738c223eec788521e904462390daceb WatchSource:0}: Error finding container d0c8d72113209c3ea0abb4d22ce2ceb9c738c223eec788521e904462390daceb: Status 404 returned error can't find the container with id d0c8d72113209c3ea0abb4d22ce2ceb9c738c223eec788521e904462390daceb Nov 01 22:37:53.445432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.445392 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:53.445535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.445514 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:53.445601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:53.445557 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:37:54.314880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:54.314825 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" event={"ID":"52ec7b7a-a00c-4206-98b1-034896709f93","Type":"ContainerStarted","Data":"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68"} Nov 01 22:37:54.315357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:54.314906 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" event={"ID":"52ec7b7a-a00c-4206-98b1-034896709f93","Type":"ContainerStarted","Data":"d0c8d72113209c3ea0abb4d22ce2ceb9c738c223eec788521e904462390daceb"} Nov 01 22:37:55.149181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:55.149150 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:37:55.153157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:55.153125 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 22:37:55.153411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:55.153380 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 22:37:56.147598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:56.147150 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 22:37:56.147598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:56.147223 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 22:37:57.151880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:57.148302 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 22:37:57.151880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:57.148389 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 22:37:58.148296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:58.148262 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 22:37:58.148599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:58.148568 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 22:37:59.147240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:59.147204 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 22:37:59.147975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:37:59.147921 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 22:38:00.198142 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:00.198104 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:38:00.198999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:00.198971 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:38:00.199728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:00.199705 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:38:00.227474 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:00.227414 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podStartSLOduration=8.227395029 podStartE2EDuration="8.227395029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:54.349836651 +0000 UTC m=+844.125823482" watchObservedRunningTime="2026-11-01 22:38:00.227395029 +0000 UTC m=+850.003381862" Nov 01 22:38:08.207751 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.207717 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 22:38:08.211975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.211954 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.234541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.234509 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 22:38:08.238929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.238904 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:38:08.243346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.243327 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.268417 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.268380 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:38:08.271953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.271925 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272160 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272302 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272430 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftb2s\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272541 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272695 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.272917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272898 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273016 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272933 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273016 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272962 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.273016 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.272991 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273031 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273073 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273109 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273144 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273196 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7796d\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273238 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273262 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.273612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.273301 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374221 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftb2s\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374265 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374289 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374321 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374346 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374371 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374400 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374434 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374469 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374501 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374550 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374605 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7796d\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374647 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374670 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374706 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374740 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.374806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374754 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374813 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374832 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374898 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374911 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.375147 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.375249 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.374809 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.375678 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.375423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375678 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.375568 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.375791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.375740 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.376159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.376075 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.377241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.377225 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.377965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.377940 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.378043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.378028 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.378242 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.378221 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.386236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.386204 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftb2s\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.386314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.386258 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.386314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.386260 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.386388 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.386306 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7796d\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.515944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.515825 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 22:38:08.522294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.521496 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.533191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.533138 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:08.548199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.548152 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 22:38:08.577606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.576564 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:08.583109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583043 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583135 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583201 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583235 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583285 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583316 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583384 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583446 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426cx\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.583288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.583495 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685554 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685611 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685650 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685711 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685763 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426cx\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.685903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685814 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.686691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685918 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.686691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.685988 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.686691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.686022 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.686691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.686550 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.687024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.686774 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.688260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.688236 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.689633 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.689012 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.689633 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.689318 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.692742 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.692404 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.695577 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.695550 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.696692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.696613 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.704328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.704244 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-426cx\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.843184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.843108 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:08.850904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.850879 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 22:38:08.856067 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:08.856036 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0a6d38_b55b_45aa_936b_609ed3615898.slice/crio-41d944c368269514ae27e252fb9a86bdf185827360850e8e44238b4ab72f5cbc WatchSource:0}: Error finding container 41d944c368269514ae27e252fb9a86bdf185827360850e8e44238b4ab72f5cbc: Status 404 returned error can't find the container with id 41d944c368269514ae27e252fb9a86bdf185827360850e8e44238b4ab72f5cbc Nov 01 22:38:08.858511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.858465 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:08.858966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.858937 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:08.859068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.858995 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:08.947750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.947664 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:38:08.948507 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:08.948470 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03acb1a5_09b5_4b7d_b213_69eb1beb342e.slice/crio-b95b137369c41d51a6f62b2d030d17de54b2f482e467ce4af3456175199afe9f WatchSource:0}: Error finding container b95b137369c41d51a6f62b2d030d17de54b2f482e467ce4af3456175199afe9f: Status 404 returned error can't find the container with id b95b137369c41d51a6f62b2d030d17de54b2f482e467ce4af3456175199afe9f Nov 01 22:38:08.951701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.951663 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:08.951798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.951742 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:08.951798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:08.951786 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:09.140818 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:09.140775 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ceb1464_83a9_4818_b3e6_963dcef48de5.slice/crio-1b4474174e140047d245e09b850c81dd74d022c4045768b0c0c8ca9fc0a327bc WatchSource:0}: Error finding container 1b4474174e140047d245e09b850c81dd74d022c4045768b0c0c8ca9fc0a327bc: Status 404 returned error can't find the container with id 1b4474174e140047d245e09b850c81dd74d022c4045768b0c0c8ca9fc0a327bc Nov 01 22:38:09.141028 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.140992 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 22:38:09.144288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.143772 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:09.144288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.143866 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:09.144288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.143913 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:09.509825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.509784 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" event={"ID":"03acb1a5-09b5-4b7d-b213-69eb1beb342e","Type":"ContainerStarted","Data":"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be"} Nov 01 22:38:09.510354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.509885 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" event={"ID":"03acb1a5-09b5-4b7d-b213-69eb1beb342e","Type":"ContainerStarted","Data":"b95b137369c41d51a6f62b2d030d17de54b2f482e467ce4af3456175199afe9f"} Nov 01 22:38:09.512359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.512330 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" event={"ID":"9a0a6d38-b55b-45aa-936b-609ed3615898","Type":"ContainerStarted","Data":"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe"} Nov 01 22:38:09.512487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.512372 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" event={"ID":"9a0a6d38-b55b-45aa-936b-609ed3615898","Type":"ContainerStarted","Data":"41d944c368269514ae27e252fb9a86bdf185827360850e8e44238b4ab72f5cbc"} Nov 01 22:38:09.515274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.515244 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" event={"ID":"7ceb1464-83a9-4818-b3e6-963dcef48de5","Type":"ContainerStarted","Data":"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d"} Nov 01 22:38:09.516191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.516169 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" event={"ID":"7ceb1464-83a9-4818-b3e6-963dcef48de5","Type":"ContainerStarted","Data":"1b4474174e140047d245e09b850c81dd74d022c4045768b0c0c8ca9fc0a327bc"} Nov 01 22:38:09.537877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.535115 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podStartSLOduration=1.5350944069999999 podStartE2EDuration="1.535094407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.533668697 +0000 UTC m=+859.309655527" watchObservedRunningTime="2026-11-01 22:38:09.535094407 +0000 UTC m=+859.311081237" Nov 01 22:38:09.540087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.540042 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:09.540372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.540331 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:09.541416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.541348 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:09.579874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.573401 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podStartSLOduration=1.573384446 podStartE2EDuration="1.573384446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.569899905 +0000 UTC m=+859.345886731" watchObservedRunningTime="2026-11-01 22:38:09.573384446 +0000 UTC m=+859.349371278" Nov 01 22:38:09.584552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.584524 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:09.596268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.596226 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podStartSLOduration=1.5962100879999999 podStartE2EDuration="1.596210088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.591043793 +0000 UTC m=+859.367030635" watchObservedRunningTime="2026-11-01 22:38:09.596210088 +0000 UTC m=+859.372196917" Nov 01 22:38:09.844933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:09.843945 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:10.589993 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.589960 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.600971 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.600921 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:10.612081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.591982 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.612081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.612021 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:10.854959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.847970 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.854959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:10.848046 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:11.542875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.534376 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.542875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.534458 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:11.584878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.584366 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.584878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.584461 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:11.844937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.844359 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.844937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:11.844436 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:12.556878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.556249 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.556878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.556319 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:12.586875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.586124 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.586875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.586181 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:12.845327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.845233 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.846439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:12.846408 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:13.541278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.541242 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.541603 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.541556 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:13.607537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.603288 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.705264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.705220 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:13.920062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.920006 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.920717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:13.920658 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:14.544878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.540828 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.544878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.540915 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:14.584281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.584243 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.584933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.584524 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:14.846938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.846494 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.846938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:14.846552 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:15.536878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.536752 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.536878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.536837 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:15.585879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.584926 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.585879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.584996 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:15.855935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.855544 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.855935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:15.855607 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:16.262412 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.262378 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 22:38:16.275106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.275079 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.280436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.280151 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 22:38:16.292227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.292199 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"distinct-w3-6ac182-kserve-self-signed-certs\"" Nov 01 22:38:16.495420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.495387 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.495681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.495659 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.495843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.495828 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.496266 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.496247 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.497128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.496379 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnndh\" (UniqueName: \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.497405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.497384 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.550750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.550658 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:38:16.557734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.557679 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.589328 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.589400 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600061 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600123 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600202 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600235 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600287 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnndh\" (UniqueName: \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.602927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.600434 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.604587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.604562 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.605693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.605215 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.605874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.605620 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.631252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.629891 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.631252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.630484 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.631252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.631182 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnndh\" (UniqueName: \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\") pod \"distinct-w3-6ac182-kserve-579d65c59c-5f76j\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.714958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.714920 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:16.939055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.933003 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:16.939055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:16.933068 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 22:38:17.299035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.298961 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 22:38:17.894931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.889308 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:17.894931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.890870 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:17.894931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.891711 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 22:38:17.980878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.972812 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:17.980878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.974018 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:17.980878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.979123 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 22:38:18.000897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.996589 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:18.000897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:17.998002 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:18.029874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:18.009180 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:38:18.032136 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:18.032041 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" event={"ID":"c560e2a1-64d9-4270-ba56-bd2217f2a687","Type":"ContainerStarted","Data":"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc"} Nov 01 22:38:18.032136 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:18.032092 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" event={"ID":"c560e2a1-64d9-4270-ba56-bd2217f2a687","Type":"ContainerStarted","Data":"deee3a1b06bc6db539c53332a80ba36ff7cf912304d5622504267fa350f624b0"} Nov 01 22:38:18.258877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:18.238298 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podStartSLOduration=2.238279908 podStartE2EDuration="2.238279908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:18.237211287 +0000 UTC m=+868.013198116" watchObservedRunningTime="2026-11-01 22:38:18.238279908 +0000 UTC m=+868.014266738" Nov 01 22:38:18.716877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:18.716035 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:19.009986 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:38:19.009090 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:38:23.759464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.759430 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:38:23.763972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.763951 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.780965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.780942 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:38:23.798108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.798075 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 22:38:23.821929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.819188 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.830760 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.830733 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 22:38:23.850605 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850581 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850617 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpj5\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850640 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850670 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850741 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850769 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850802 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850817 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.850928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.850835 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.952403 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.952585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952421 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.952585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952547 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bl5v\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.952690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952588 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.952690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952653 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.952777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952691 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.952777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952739 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.952895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952782 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.952895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952821 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.952895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952870 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.953054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952906 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.953054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952961 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.953054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.952990 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.953054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953034 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.953272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953070 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpj5\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.953272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953097 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:23.953272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953109 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.953272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953134 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.953650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.953487 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.954082 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.954108 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.954310 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.954623 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.956023 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.956761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.956656 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.978504 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.978482 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpj5\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:23.986542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:23.986516 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:24.055809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.055737 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.055957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.055821 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl5v\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.055957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.055901 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.055953 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.055999 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056059 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056224 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056086 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056224 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056183 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056291 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056349 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056426 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056383 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.056663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056642 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.057014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.056983 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.057531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.057473 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.058997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.058973 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.059553 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.059536 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.064209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.064185 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.065322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.065300 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl5v\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.078280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.078264 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:24.148225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.148195 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:24.372434 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:24.372403 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5254fd0a_d17d_448d_8195_7adfa483b6a7.slice/crio-0cfe756226c72bb585ed2b25db4b05f6eef87d82c8a3c9fe55fe75dda769f473 WatchSource:0}: Error finding container 0cfe756226c72bb585ed2b25db4b05f6eef87d82c8a3c9fe55fe75dda769f473: Status 404 returned error can't find the container with id 0cfe756226c72bb585ed2b25db4b05f6eef87d82c8a3c9fe55fe75dda769f473 Nov 01 22:38:24.374975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.374934 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:24.375089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.375017 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:24.375089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.375082 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:24.376513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.376491 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:38:24.463604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.463571 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 22:38:24.464905 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:24.464772 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d391df6_8e9d_428e_af18_b093f11dc5c2.slice/crio-126e982f104d2f506813b405a44c2802728bb9320a26bcfe67e43ca5e2882328 WatchSource:0}: Error finding container 126e982f104d2f506813b405a44c2802728bb9320a26bcfe67e43ca5e2882328: Status 404 returned error can't find the container with id 126e982f104d2f506813b405a44c2802728bb9320a26bcfe67e43ca5e2882328 Nov 01 22:38:24.469642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.469613 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:24.469740 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.469679 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:24.469740 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:24.469708 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:25.360004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.359958 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" event={"ID":"5254fd0a-d17d-448d-8195-7adfa483b6a7","Type":"ContainerStarted","Data":"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32"} Nov 01 22:38:25.360651 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.360624 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" event={"ID":"5254fd0a-d17d-448d-8195-7adfa483b6a7","Type":"ContainerStarted","Data":"0cfe756226c72bb585ed2b25db4b05f6eef87d82c8a3c9fe55fe75dda769f473"} Nov 01 22:38:25.376367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.376322 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" event={"ID":"4d391df6-8e9d-428e-af18-b093f11dc5c2","Type":"ContainerStarted","Data":"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f"} Nov 01 22:38:25.376673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.376635 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" event={"ID":"4d391df6-8e9d-428e-af18-b093f11dc5c2","Type":"ContainerStarted","Data":"126e982f104d2f506813b405a44c2802728bb9320a26bcfe67e43ca5e2882328"} Nov 01 22:38:25.446427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.446366 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podStartSLOduration=2.44634445 podStartE2EDuration="2.44634445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:25.407011135 +0000 UTC m=+875.183011249" watchObservedRunningTime="2026-11-01 22:38:25.44634445 +0000 UTC m=+875.222331280" Nov 01 22:38:25.448149 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:25.448095 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podStartSLOduration=2.448083229 podStartE2EDuration="2.448083229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:25.44433337 +0000 UTC m=+875.220320195" watchObservedRunningTime="2026-11-01 22:38:25.448083229 +0000 UTC m=+875.224070059" Nov 01 22:38:26.060829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.060779 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 22:38:26.076202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.076166 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.080089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.079954 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:26.082456 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.082428 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 22:38:26.082800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.082779 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.083021 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.082962 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:26.110026 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.108051 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xhx7b\"" Nov 01 22:38:26.110026 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.108903 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 22:38:26.168626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.168590 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:26.178870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.178818 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.179125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.179095 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:26.201198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.201168 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 22:38:26.201418 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.201395 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 22:38:26.214189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214163 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214200 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214257 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214305 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214347 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92dc\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214365 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.214468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.214400 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.339928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.326459 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340180 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340379 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340365 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340537 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340684 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92dc\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340787 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.340949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.340935 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.348049 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.343919 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.391364 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.391331 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.430650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.430617 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92dc\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.492905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.473251 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.492905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.491803 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.518875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.514589 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.551920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.523834 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8lx4\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.551920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.525618 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:26.728957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.728918 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:26.807191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:26.807160 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:27.075876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.072233 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 22:38:27.080080 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.079935 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.080080 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:27.079973 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a148739_f6ef_4959_b862_beb7b3ea7aea.slice/crio-677dd39f01a093b8bdc7b838d4b7b3d24ea868fd049bb453ab7fe0cc875315f5 WatchSource:0}: Error finding container 677dd39f01a093b8bdc7b838d4b7b3d24ea868fd049bb453ab7fe0cc875315f5: Status 404 returned error can't find the container with id 677dd39f01a093b8bdc7b838d4b7b3d24ea868fd049bb453ab7fe0cc875315f5 Nov 01 22:38:27.080080 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.080006 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:27.169900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.162316 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.169900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.162375 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:27.585835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.585790 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" event={"ID":"8a148739-f6ef-4959-b862-beb7b3ea7aea","Type":"ContainerStarted","Data":"677dd39f01a093b8bdc7b838d4b7b3d24ea868fd049bb453ab7fe0cc875315f5"} Nov 01 22:38:27.600284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:27.600257 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:28.082252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:28.082216 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.082528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:28.082495 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:28.150502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:28.150460 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.150770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:28.150743 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:29.085879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:29.082056 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.085879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:29.082117 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:29.151410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:29.150938 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.151410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:29.151002 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:30.082692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:30.082243 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.082692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:30.082305 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:30.149624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:30.149203 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.149624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:30.149262 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:31.084172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:31.084136 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.084569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:31.084535 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 22:38:31.155128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:31.155091 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.155605 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:31.155187 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 01 22:38:32.286873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.278596 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:32.288935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.288716 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:32.297758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.297704 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 22:38:32.487925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.487877 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:32.489510 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.489456 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:32.491664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:32.491638 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:38:33.399211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:33.399173 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:33.399617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:33.399257 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:33.663368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:33.663333 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" event={"ID":"8a148739-f6ef-4959-b862-beb7b3ea7aea","Type":"ContainerStarted","Data":"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8"} Nov 01 22:38:33.664030 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:33.663996 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:33.689554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:33.689505 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" podStartSLOduration=1.373453989 podStartE2EDuration="7.689487461s" totalImagesPullingTime="6.316033472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:33.684542799 +0000 UTC m=+883.460529651" watchObservedRunningTime="2026-11-01 22:38:33.689487461 +0000 UTC m=+883.465474289" Nov 01 22:38:34.673045 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:34.673013 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 22:38:38.089220 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.089179 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:38:38.094120 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.094093 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.098622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.098603 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e5f47e-openshift-default-dockercfg-lfvrz\"" Nov 01 22:38:38.102392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102365 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102406 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102434 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbk9n\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102579 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102632 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102664 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102698 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102743 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.102792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.102772 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.110083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.110064 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:38:38.204025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.203988 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204029 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204311 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204055 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbk9n\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204311 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204086 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204311 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204119 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204474 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204378 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204534 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204481 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204593 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204546 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204600 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204628 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204691 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.204766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.204744 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.205056 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.205031 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.205164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.205102 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.207323 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.207297 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.207433 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.207391 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.219173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.219147 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.219294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.219264 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbk9n\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.407418 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.407394 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:38.713267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.713187 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:38:38.717120 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:38:38.717089 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004d2649_2fc7_49ca_a1ab_489fb6f12218.slice/crio-24b0b8869bc5380a94b102fcb517e17d6c4b246248f64128b88f4b36e6f1f097 WatchSource:0}: Error finding container 24b0b8869bc5380a94b102fcb517e17d6c4b246248f64128b88f4b36e6f1f097: Status 404 returned error can't find the container with id 24b0b8869bc5380a94b102fcb517e17d6c4b246248f64128b88f4b36e6f1f097 Nov 01 22:38:38.720159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.720117 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:38.720276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.720206 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:38.720276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:38.720249 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:38:39.711246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:39.711204 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" event={"ID":"004d2649-2fc7-49ca-a1ab-489fb6f12218","Type":"ContainerStarted","Data":"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979"} Nov 01 22:38:39.711644 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:39.711273 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" event={"ID":"004d2649-2fc7-49ca-a1ab-489fb6f12218","Type":"ContainerStarted","Data":"24b0b8869bc5380a94b102fcb517e17d6c4b246248f64128b88f4b36e6f1f097"} Nov 01 22:38:39.745873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:39.745791 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podStartSLOduration=1.745771001 podStartE2EDuration="1.745771001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:39.737665058 +0000 UTC m=+889.513651892" watchObservedRunningTime="2026-11-01 22:38:39.745771001 +0000 UTC m=+889.521757832" Nov 01 22:38:40.407824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:40.407794 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:40.409267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:40.409245 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.409384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:40.409310 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:41.407884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:41.407831 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.408372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:41.407928 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:42.407930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:42.407895 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.408322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:42.407955 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:43.407888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:43.407842 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.408058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:43.407908 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:44.407804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:44.407766 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.408014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:44.407825 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:45.408717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:45.408681 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.409305 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:45.408742 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 22:38:47.425877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.420953 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:47.425877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.421015 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.590585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.589973 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:38:47.689075 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.682216 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:47.689075 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.684371 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:47.728895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.720462 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:38:47.962618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:47.961456 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.350071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:48.336126 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:38:48.594839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:48.594782 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.674915 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:48.671516 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:48.674915 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:48.671602 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.735877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:48.734895 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:49.122980 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:38:49.122872 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:38:49.131927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:49.126437 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.131927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:49.126490 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.131927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:49.127247 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:38:49.131927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:49.120991 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:38:50.506595 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:50.506567 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:38:50.507091 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:38:50.506815 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:39:07.968258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:07.968182 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:39:07.983495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:07.980440 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.025144 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.032813 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.032877 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.032908 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.032932 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.033005 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.033087 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkz9z\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.033155 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.033256 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.033908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.033279 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.101976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.101862 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:39:08.106974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.106951 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.121018 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.120890 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136417 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136489 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136575 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz9z\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136680 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136741 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136769 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136911 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136950 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.136978 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137013 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cq2x\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137062 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137123 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137156 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137230 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137256 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137344 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137370 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.137433 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.138314 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.139518 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.139870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.139773 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.150084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.139927 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.150084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.140205 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.150084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.143044 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.150084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.143191 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.150084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.144519 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 22:39:08.154393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.154376 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.171876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.171834 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz9z\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.176008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.175980 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 22:39:08.176234 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.176213 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238461 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.238535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238501 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238530 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238581 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238620 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238648 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vjg\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238668 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238683 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238729 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238746 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.238790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238763 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238830 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238868 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cq2x\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238886 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238900 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238938 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.238967 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.239241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.239031 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.239596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.239331 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.239596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.239554 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.240614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.240592 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.240984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.240957 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.241181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.241151 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.242274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.242244 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.242549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.242532 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.250721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.250697 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cq2x\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.250913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.250894 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.340117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340087 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340170 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340188 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340226 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340306 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340356 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340393 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vjg\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340423 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340439 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.340931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.340876 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.341126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.341103 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.341221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.341202 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.341423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.341387 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.341423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.341417 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.341820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.341771 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:08.343425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.343404 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.343709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.343688 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.350106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.350078 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.350534 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.350512 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vjg\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.435716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.435688 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:08.506948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.506108 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:08.643115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.643082 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:39:08.651549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.650639 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.651549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.650695 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.651549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.650723 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.666790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.666741 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:39:08.666942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.666798 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:39:08.667013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.666961 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748260 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748414 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748473 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748586 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hj7\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748612 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748631 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748751 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748733 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748795 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748767 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.748837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.748809 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.777345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.777322 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:39:08.780330 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:39:08.780307 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6ebf20_b0bd_4afe_8a67_de7f357db9f8.slice/crio-20fc53d1ffd380a6f95fcdc61b2978942830e72e0e785d2902e1fd40f6c82e82 WatchSource:0}: Error finding container 20fc53d1ffd380a6f95fcdc61b2978942830e72e0e785d2902e1fd40f6c82e82: Status 404 returned error can't find the container with id 20fc53d1ffd380a6f95fcdc61b2978942830e72e0e785d2902e1fd40f6c82e82 Nov 01 22:39:08.783587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.783560 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.783670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.783619 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.783670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.783649 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.844939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.840702 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.844939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.840791 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.844939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.840879 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:08.844939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.844068 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855040 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855125 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855164 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855200 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hj7\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855219 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855242 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855285 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855307 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.855660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.855347 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.857686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.857100 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.857686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.857345 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.858973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.858481 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.858973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.858727 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.858973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.858941 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.861725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.861657 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.862546 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.862522 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.873325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.873305 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:08.873442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:08.873390 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hj7\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:09.006262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.004478 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:09.387770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.387742 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:39:09.388352 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:39:09.388327 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e70486d_59c4_4b4c_bcb3_317e7f1e7cb4.slice/crio-fa53494555338288dc6fd800df9e927a47fa4c9d1d0ab1c25ebf122537ab37f7 WatchSource:0}: Error finding container fa53494555338288dc6fd800df9e927a47fa4c9d1d0ab1c25ebf122537ab37f7: Status 404 returned error can't find the container with id fa53494555338288dc6fd800df9e927a47fa4c9d1d0ab1c25ebf122537ab37f7 Nov 01 22:39:09.391486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.391449 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:09.391579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.391528 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:09.391579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.391567 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:09.467120 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.467070 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" event={"ID":"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4","Type":"ContainerStarted","Data":"fa53494555338288dc6fd800df9e927a47fa4c9d1d0ab1c25ebf122537ab37f7"} Nov 01 22:39:09.469258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.468976 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" event={"ID":"afe9f559-e843-40c3-901d-fde2eba8250b","Type":"ContainerStarted","Data":"af7b77d4f6f71f75d2da71c725fbc5160a61f89f2d9193078606e2fa0323ddb2"} Nov 01 22:39:09.469258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.469027 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" event={"ID":"afe9f559-e843-40c3-901d-fde2eba8250b","Type":"ContainerStarted","Data":"69e28b000ec9649046ab6027159b9d84bce36125e762d9426bc0757bdaca4c7c"} Nov 01 22:39:09.470998 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.470977 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" event={"ID":"565eebb6-d87e-4f1c-bae0-e78d741850d4","Type":"ContainerStarted","Data":"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64"} Nov 01 22:39:09.471102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.471022 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" event={"ID":"565eebb6-d87e-4f1c-bae0-e78d741850d4","Type":"ContainerStarted","Data":"8241e7c98a17eb0a5a1340e955575b9fbdc234f041440669364b6a318bcd8bcb"} Nov 01 22:39:09.472788 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.472764 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" event={"ID":"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8","Type":"ContainerStarted","Data":"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1"} Nov 01 22:39:09.472950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.472810 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" event={"ID":"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8","Type":"ContainerStarted","Data":"20fc53d1ffd380a6f95fcdc61b2978942830e72e0e785d2902e1fd40f6c82e82"} Nov 01 22:39:09.491395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.491345 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podStartSLOduration=1.491332021 podStartE2EDuration="1.491332021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.488990481 +0000 UTC m=+919.264977313" watchObservedRunningTime="2026-11-01 22:39:09.491332021 +0000 UTC m=+919.267318904" Nov 01 22:39:09.507889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.507864 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:09.519575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:09.519052 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podStartSLOduration=1.5190338410000002 podStartE2EDuration="1.519033841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.514006431 +0000 UTC m=+919.289993260" watchObservedRunningTime="2026-11-01 22:39:09.519033841 +0000 UTC m=+919.295020669" Nov 01 22:39:10.349125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.349091 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:10.350781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.350755 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.351037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.351011 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:10.456033 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.455991 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.456323 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.456296 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:10.469875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.467027 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:10.553714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.553676 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.554011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.553979 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:10.772519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.772480 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" event={"ID":"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4","Type":"ContainerStarted","Data":"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867"} Nov 01 22:39:10.835879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.823800 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podStartSLOduration=2.8237805700000003 podStartE2EDuration="2.82378057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:10.822452723 +0000 UTC m=+920.598439587" watchObservedRunningTime="2026-11-01 22:39:10.82378057 +0000 UTC m=+920.599767405" Nov 01 22:39:10.835879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:10.825264 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podStartSLOduration=3.825251197 podStartE2EDuration="3.825251197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.545135525 +0000 UTC m=+919.321122368" watchObservedRunningTime="2026-11-01 22:39:10.825251197 +0000 UTC m=+920.601238025" Nov 01 22:39:11.033941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.017481 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:11.057971 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.034269 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.082250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.082207 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:11.095107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.095069 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.325931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.320930 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.442293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.442257 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.456397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.456359 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:11.457173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.457136 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.473749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.473716 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:11.509962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.509931 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.510242 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.510219 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:11.950886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:11.948160 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.015272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.015232 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.015550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.015520 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:12.067878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.067024 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.101145 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.101102 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.101747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.101720 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:12.426874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.418661 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.426874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.418743 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:12.524877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.491296 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.524877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.491354 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:12.524877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.511044 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.524877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.511095 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:12.950501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:12.948961 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.029877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.028357 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.029877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.028415 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:13.037333 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.037303 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:13.037581 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.037566 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:39:13.347018 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.346925 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.347269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.347236 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:13.368270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.368227 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:13.445013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.444978 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.445284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.445257 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:13.517887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.508746 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.524887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.518149 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:13.772032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:13.771979 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:39:14.006015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.005968 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.007599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.007573 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.007792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.007769 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:14.089149 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.089035 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.343933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.343696 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.343933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.343755 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:14.371962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.371923 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.405501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.405470 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:14.440695 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.440661 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.448440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.448028 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:14.651205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.651010 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.651205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.651100 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:14.738583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.738558 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:14.799101 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:14.799053 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.066203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.015971 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.196969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.039771 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.197279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.197240 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:15.197451 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.124089 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.197907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.115304 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:15.198158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.198142 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:15.199463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.159591 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.375579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.375486 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.375838 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.375808 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:15.442063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.441997 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.442569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.442509 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:15.596275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.596229 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.596522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.596491 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:15.790912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.790564 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.967877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:15.960680 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:16.020175 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.020144 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.020951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.020435 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:16.173749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.173706 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 01 22:39:16.189956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.189918 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.199726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.199690 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.227319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.227274 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:16.237144 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.237108 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:39:16.432080 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.431980 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.432327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.432298 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:16.446267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.446237 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.446459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.446436 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:16.529883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.521558 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.529883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.521623 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:16.792604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.792511 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.793870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:16.793824 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:17.071211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.070464 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.071211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.070519 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:17.193027 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.192981 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.202653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.202579 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.255092 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.255021 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.346113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.346017 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.346364 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.346337 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:17.466896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.441205 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.466896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.441296 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:17.509287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.509253 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.559009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.558957 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:17.732878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.732381 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.796047 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:17.795983 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.018933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.018229 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.018933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.018291 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:18.072126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.070969 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.197107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.195985 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.206287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.206229 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.277659 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.277559 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.342116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.341979 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.342116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.342047 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.394799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.394765 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.395098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.395066 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:18.438420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.438387 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.438674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.438647 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:18.510373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.510319 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.510681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.510636 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:18.549114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.548988 5778 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.549114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.549086 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.733148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.733112 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.733393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.733367 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.814123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.814026 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.946461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.946422 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.946785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:18.946738 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.005221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.005183 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.005522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.005493 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:19.032321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.032289 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:19.032638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.032596 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.053487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.053454 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:19.053782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.053734 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.073072 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.072959 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.074461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.074440 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:19.122344 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:39:19.122269 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:39:19.208877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.207124 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:19.210588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.210463 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:19.285792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.285736 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:19.344604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.344502 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.344946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.344897 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:19.436878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.436681 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.436878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.436738 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:19.514879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.510235 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.514879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.510299 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:19.909885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.819102 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:20.015878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:19.978977 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.128880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.017885 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.154008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.132174 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:20.154008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.091625 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.291109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.290968 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.307884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.293081 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.309627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.303036 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:20.413877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.397760 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.414402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.414338 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:20.601248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.601150 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.601520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.601478 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:20.606829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.601719 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.638233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.638180 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:20.835876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:20.829060 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:21.005201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.005164 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.005784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.005751 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:21.135089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.135042 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.292083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.291961 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:21.313074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.311941 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.315636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.314949 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:21.344690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.344235 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.344690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.344295 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:21.440879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.440009 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.440879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.440092 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:21.513774 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.513742 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.514063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.514036 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:21.829995 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:21.829946 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.035918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.015977 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.037046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.036998 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:22.208750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.205976 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.294883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.293952 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.319887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.314939 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.320167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.320138 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:22.352127 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.352089 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.352369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.352339 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:22.437454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.437419 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.437839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.437723 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:22.526083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.525983 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.539132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.538983 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:22.709520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:22.709491 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:23.010250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.010209 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.035989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.035945 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:23.248976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.248915 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.294829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.294708 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.318668 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.318626 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.324896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.324840 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.343990 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.343962 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:23.345769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.345730 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.346025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.346000 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:23.411894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.411219 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:23.441093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.441058 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:23.443617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.443592 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.443802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.443778 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:23.523521 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.523471 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:23.524722 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.524694 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.524985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.524941 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:23.828875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:23.823146 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.007993 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.007010 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:24.022874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.014750 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.022874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.014803 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:24.115176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.114957 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:39:24.193667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.193559 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:24.335505 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.335472 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:39:24.374930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.343366 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.376604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.376571 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 22:39:24.456878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.456692 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.456878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.456766 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:24.508642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.507520 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.508642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:24.507576 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 22:39:25.005563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:25.005532 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:25.005802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:25.005774 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 22:39:25.444896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:25.440411 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:39:25.444896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:25.440474 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 22:39:26.348586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.348181 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:26.348586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.348242 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:26.494693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.493996 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:26.497287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.497213 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 22:39:26.519628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.519592 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:26.529316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.529288 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:39:26.703831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.703789 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:26.708109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:26.708074 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:39:27.398032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:27.397990 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:27.398238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:27.398044 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:28.130000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:28.129964 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:28.178946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:28.178916 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:39:38.253430 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.253395 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:39:38.257821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.257797 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.272824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.272794 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:39:38.303193 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.303137 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 22:39:38.309143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.309110 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.334111 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.333459 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 22:39:38.386226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386193 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386316 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386346 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386400 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386445 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386478 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386528 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386549 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4mc\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386568 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386671 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386708 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386770 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386794 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.386884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386811 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386885 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.386957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.386925 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.387021 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.387005 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.387082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.387065 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.488543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488503 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.488543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488542 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488560 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488584 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488621 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488649 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488673 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4mc\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.488758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488703 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488764 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488798 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488925 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488956 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.488981 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489025 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489158 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489234 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489319 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489350 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489361 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489112 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489485 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489664 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489720 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489760 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.489831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489076 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.489961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.489826 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.491002 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.490981 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.491152 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.491007 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.492141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.492115 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.492552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.492532 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.493602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.493580 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.499674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.499639 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.499928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.499906 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.504902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.503015 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4mc\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.504902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.504481 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.504902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.504494 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.575112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.575072 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:38.644560 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.644306 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:38.880514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.880490 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:39:38.881195 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:39:38.881172 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7b01d1_72f2_44e6_8ea2_dc292e43fffd.slice/crio-853726c2affe46d0773c48e60e1ff6026d21b65cdd3a376c401d4424360b076c WatchSource:0}: Error finding container 853726c2affe46d0773c48e60e1ff6026d21b65cdd3a376c401d4424360b076c: Status 404 returned error can't find the container with id 853726c2affe46d0773c48e60e1ff6026d21b65cdd3a376c401d4424360b076c Nov 01 22:39:38.883583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.883551 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:38.883665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.883616 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:38.883665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.883644 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:38.992398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.992364 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 22:39:38.992551 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:39:38.992504 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8bc47b_2368_4ad3_b00b_252c7118855a.slice/crio-5755f59278838b60eb01d58b8e7b2dd2b2138085994d6823b9eadeaab7199441 WatchSource:0}: Error finding container 5755f59278838b60eb01d58b8e7b2dd2b2138085994d6823b9eadeaab7199441: Status 404 returned error can't find the container with id 5755f59278838b60eb01d58b8e7b2dd2b2138085994d6823b9eadeaab7199441 Nov 01 22:39:38.995601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.995543 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:38.995680 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.995655 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:38.995744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:38.995704 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.085302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.085265 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" event={"ID":"2a8bc47b-2368-4ad3-b00b-252c7118855a","Type":"ContainerStarted","Data":"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381"} Nov 01 22:39:39.085470 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.085336 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" event={"ID":"2a8bc47b-2368-4ad3-b00b-252c7118855a","Type":"ContainerStarted","Data":"5755f59278838b60eb01d58b8e7b2dd2b2138085994d6823b9eadeaab7199441"} Nov 01 22:39:39.087528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.087507 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" event={"ID":"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd","Type":"ContainerStarted","Data":"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f"} Nov 01 22:39:39.087677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.087665 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" event={"ID":"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd","Type":"ContainerStarted","Data":"853726c2affe46d0773c48e60e1ff6026d21b65cdd3a376c401d4424360b076c"} Nov 01 22:39:39.108929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.108845 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:39:39.112727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.112672 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podStartSLOduration=1.112657824 podStartE2EDuration="1.112657824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.108135296 +0000 UTC m=+948.884122128" watchObservedRunningTime="2026-11-01 22:39:39.112657824 +0000 UTC m=+948.888644667" Nov 01 22:39:39.114965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.114924 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.128884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.128834 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:39:39.151811 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.151734 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podStartSLOduration=1.151713116 podStartE2EDuration="1.151713116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.145363521 +0000 UTC m=+948.921350351" watchObservedRunningTime="2026-11-01 22:39:39.151713116 +0000 UTC m=+948.927699946" Nov 01 22:39:39.158872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.158828 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 22:39:39.182304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.181659 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.194752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.192783 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.202880 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.202949 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203022 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203080 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203123 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203166 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203280 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203363 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6jk\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.203926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.203408 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305281 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305335 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpxg\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305398 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305429 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305532 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305623 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305654 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305725 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.305923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305823 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305925 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.305740 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306145 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306192 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.306257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306239 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.306554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306539 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306622 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306704 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306624 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6jk\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.306704 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306690 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.306800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306723 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.306800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306761 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.306987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.306794 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.307073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.307053 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.308212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.308180 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.308536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.308520 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.316288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.316260 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6jk\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.316488 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.316468 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.408642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408605 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.408835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408693 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpxg\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.408835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408721 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.408835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408748 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.408835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408812 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.408835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408836 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408881 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408907 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.408939 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.409156 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.409193 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409493 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.409452 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.409540 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.409760 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.409743 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.411562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.411537 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.411800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.411782 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.418699 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.418669 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpxg\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.418890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.418706 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.449754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.449715 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:39.522878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.516909 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:39.601327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.592792 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:39.601327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.595016 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:39.601327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.595071 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:39.645458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.645356 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:39.809386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.809355 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:39:39.811821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.811791 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.811981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.811912 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.811981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.811962 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.962947 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.954778 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 22:39:39.995114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.995050 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.995226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.995120 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:39.995226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:39.995146 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:40.098513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.098483 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" event={"ID":"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd","Type":"ContainerStarted","Data":"e94eab422ba5395c43453bb3db274e21d2bfa7c2cae6a86e0d7133ff585accb7"} Nov 01 22:39:40.100324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.100241 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" event={"ID":"5369d805-44aa-4cc2-9c18-b93525ab9b61","Type":"ContainerStarted","Data":"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a"} Nov 01 22:39:40.100324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.100287 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" event={"ID":"5369d805-44aa-4cc2-9c18-b93525ab9b61","Type":"ContainerStarted","Data":"0b03717823dddc7304cc14ead03e3c9014069064d98b26ceceb201aabee71b2c"} Nov 01 22:39:40.461513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.461426 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:40.588910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.585239 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.588910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.585299 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:40.649652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.649616 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.649982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:40.649941 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:41.109784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.109740 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" event={"ID":"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd","Type":"ContainerStarted","Data":"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23"} Nov 01 22:39:41.169876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.156335 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podStartSLOduration=2.15631691 podStartE2EDuration="2.15631691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:41.139293496 +0000 UTC m=+950.915280341" watchObservedRunningTime="2026-11-01 22:39:41.15631691 +0000 UTC m=+950.932303743" Nov 01 22:39:41.169876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.157396 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podStartSLOduration=2.157384645 podStartE2EDuration="2.157384645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:40.127449562 +0000 UTC m=+949.903436393" watchObservedRunningTime="2026-11-01 22:39:41.157384645 +0000 UTC m=+950.933371474" Nov 01 22:39:41.467876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.452644 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.467876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.452710 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:41.539888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.530452 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:41.539888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.530722 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.539888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.530767 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:41.578888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.576271 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.578888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.576329 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:41.649936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.647299 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.649936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:41.647378 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:42.462874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.462318 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.462874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.462381 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:42.529755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.529298 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.529755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.529362 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:42.582758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.582297 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.582758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.582359 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:42.647514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.647481 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.647823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.647777 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:42.859956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:42.854501 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.137081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.136948 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.209970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.209926 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:43.280626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.280582 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.455289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.455254 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.455551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.455523 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:43.528352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.528311 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.528630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.528601 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:43.588014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.587979 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.616878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.605941 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:43.648461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.648427 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.648759 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.648716 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:43.871962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:43.861920 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.222880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.220351 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.222880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.221324 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.222880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.222560 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:44.242900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.223303 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:44.242900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.223366 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:44.300423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.284094 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.301542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.301515 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:39:44.552397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.550668 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.552397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.550729 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:44.552717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.552693 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.552899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.552876 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:44.685507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.685470 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.686143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.686110 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:44.689300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.689273 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.689511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.689484 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:44.909251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.909208 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.955649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:44.955598 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:45.224041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.223933 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.259943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.257570 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:39:45.259943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.259477 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.264117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.250316 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:39:45.304309 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.301877 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.477928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.463076 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.478194 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.478161 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:45.537295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.537257 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.537571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.537543 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:45.577530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.577490 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.592925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.592887 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:45.667663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.667627 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.667957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.667925 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:45.915033 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:45.914981 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.195065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.194953 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 01 22:39:46.416892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.260461 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:46.426881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.260673 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:46.448907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.268020 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.449511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.308114 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:46.628196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.628156 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.640877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.636958 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:46.772241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.659925 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.772241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.660091 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:46.821878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.821286 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.821878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.821427 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:46.985618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:46.981195 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.020972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.010746 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.020972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.010807 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:47.423992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.423939 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.437935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.429346 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.485562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.457260 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.485562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.459937 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.485562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.460345 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.518936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.487540 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:47.566919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.561879 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.566919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.561945 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:47.615877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.582705 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.615877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.582764 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:47.707974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.707880 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.708274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.708233 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:47.851121 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.851067 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.991376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:47.991247 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:48.465610 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.465576 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.465965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.465921 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:48.493877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.488941 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:39:48.493877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.489337 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:48.493877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.489614 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:48.493877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.489886 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:48.538837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.538784 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.539087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.538870 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:48.586890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.578233 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.586890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.578291 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:48.648554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.648519 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.648831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.648804 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:48.996878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:48.995956 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.035876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.034956 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:49.035876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.035022 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.082877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.069988 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:49.082877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.070050 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.193132 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:39:49.193045 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:39:49.456292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.456256 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.456587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.456556 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:49.489288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.489243 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.490740 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.490710 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.491273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.490945 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:49.492057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.490969 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.523523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.523490 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.523820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.523781 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:49.584878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.583966 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.584878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.584053 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:49.647879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.646417 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.647879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.646499 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:49.975878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:49.974958 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.000880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.000428 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:50.503939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.494498 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.503939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.495277 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:39:50.503939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.495356 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.503939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.499400 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.562875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.553039 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.562875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.553107 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:50.563143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.562873 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.569880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.569809 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:50.581997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.581380 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.581997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.581446 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:50.667051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.667020 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.670981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.670928 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:50.797395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.796133 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:50.942029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:50.941999 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:39:51.039876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.009026 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:51.479874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.465558 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.479874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.465617 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:51.540666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.540633 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.540951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.540924 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:51.566845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.566745 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.567429 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.567386 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:39:51.601217 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.590241 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.601217 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.590318 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:51.660974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.660940 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.661239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:51.661212 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:52.012058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.011981 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:52.297948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.291006 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.349888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.345995 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.492920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.492885 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.511989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.511946 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:52.539178 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.530181 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.539701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.539662 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:52.572307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.569109 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.573543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.570078 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:52.577132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.577045 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.577132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.577087 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:52.646982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.646947 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.672514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.672474 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:52.911938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:52.902929 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 01 22:39:53.021076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.021029 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.239310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.239198 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:39:53.324239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.324207 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:39:53.391222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.391182 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.394276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.392438 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:53.436309 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.435085 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:39:53.526924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.512045 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.526924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.512109 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:53.538668 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.538599 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.538921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.538869 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:53.581987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.581959 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:53.595868 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.595817 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.596246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.596214 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 01 22:39:53.651331 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.651303 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:53.653831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.653805 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.654152 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.654127 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 01 22:39:53.847297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:53.847218 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:39:54.451420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.451386 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:54.458731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.458701 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.459014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.458985 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:54.533327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.529702 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:54.533327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.531405 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.533327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:54.531493 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:55.461875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.457015 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:55.461875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.457090 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 01 22:39:55.524779 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.524729 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:55.524991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.524794 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 22:39:55.597641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.597269 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:55.597641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.597327 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:55.649966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.649928 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:55.650238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:55.650207 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:56.581561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.581520 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:56.602116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.602087 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:39:56.681255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.677878 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:56.681255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.677942 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:56.695969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.695938 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:56.696246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:56.696218 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:57.314701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.314668 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:57.333131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.333103 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 22:39:57.362294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.362266 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:57.365926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.365899 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 22:39:57.407652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.407620 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:39:57.410376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:39:57.410347 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:40:13.239541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.239499 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-ec3f01"] Nov 01 22:40:13.245780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.245754 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:13.252807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.252784 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 22:40:13.257430 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.257405 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ec3f01"] Nov 01 22:40:13.367743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.367712 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2qq\" (UniqueName: \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\") pod \"test-curl-6968-ec3f01\" (UID: \"16d85602-8b11-416e-bd85-9f582bc86b64\") " pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:13.469439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.469401 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2qq\" (UniqueName: \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\") pod \"test-curl-6968-ec3f01\" (UID: \"16d85602-8b11-416e-bd85-9f582bc86b64\") " pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:13.480879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.480812 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2qq\" (UniqueName: \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\") pod \"test-curl-6968-ec3f01\" (UID: \"16d85602-8b11-416e-bd85-9f582bc86b64\") " pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:13.671958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:13.671845 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:14.929877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:14.927164 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:14.992833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:14.992789 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.297542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.269323 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.327876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.304929 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.366429 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.349880 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:40:15.371414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.371387 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ec3f01"] Nov 01 22:40:15.443877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.442931 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.951345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.946791 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.951345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:15.948480 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:16.043382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.015939 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.043382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.016248 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:16.228647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.226380 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.339899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.335958 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.339899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.336362 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:40:16.339899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.336678 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:40:16.339899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.336736 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:16.572253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:16.572133 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:17.216876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.182628 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.496580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.496470 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.773932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.773186 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 01 22:40:17.773932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.773329 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:17.799877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.798905 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:17.835997 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:17.835954 5778 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.47s" Nov 01 22:40:17.836200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.836021 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ec3f01" event={"ID":"16d85602-8b11-416e-bd85-9f582bc86b64","Type":"ContainerStarted","Data":"e00dd132bd0f8ad4579bfaabc2f8c209fb581c1d81830483960aa14587d0811c"} Nov 01 22:40:17.836292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.836270 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:17.836364 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.836352 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:40:17.839880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.836477 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:17.839880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:17.836761 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:18.062260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:18.062160 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:40:18.447468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:18.447423 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.612876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:18.608258 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:40:19.017877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:18.991509 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:19.066338 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.018678 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.067396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.067368 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:19.113988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.113948 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:19.114278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.114246 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.512877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.477986 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.512877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.478414 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:19.557112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.516555 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:19.557112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.516625 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.557112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.517095 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:40:19.557112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.539689 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:19.676879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.632721 5778 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:19.676879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.632790 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.702649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.678300 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:40:19.995891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:19.988687 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.066920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.041786 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.101883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.095610 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.489914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.479457 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.526261 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.526218 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.623542 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:20.623506 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:40:20.640012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.639985 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:20.640198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.640048 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:40:20.640198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.640078 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:40:20.663814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.663782 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:20.666303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:20.666269 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:22.818267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:22.818163 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ec3f01" event={"ID":"16d85602-8b11-416e-bd85-9f582bc86b64","Type":"ContainerStarted","Data":"358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c"} Nov 01 22:40:22.838211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:22.838144 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-ec3f01" podStartSLOduration=2.652553318 podStartE2EDuration="9.838126954s" totalImagesPullingTime="7.185573636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:22.835990183 +0000 UTC m=+992.611977013" watchObservedRunningTime="2026-11-01 22:40:22.838126954 +0000 UTC m=+992.614113786" Nov 01 22:40:23.533302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.533268 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-ec3f01"] Nov 01 22:40:23.539344 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.539317 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-ec3f01"] Nov 01 22:40:23.704189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.704148 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-1d5714"] Nov 01 22:40:23.705583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.705558 5778 cpu_manager.go:414] "containerMap: removing container" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" containerName="test-curl-6968-ec3f01" Nov 01 22:40:23.705583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.705584 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" containerName="test-curl-6968-ec3f01" Nov 01 22:40:23.705958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.705933 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" containerName="test-curl-6968-ec3f01" Nov 01 22:40:23.711326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.711297 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:23.717381 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.717353 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-1d5714"] Nov 01 22:40:23.778570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.778542 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k78h\" (UniqueName: \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\") pod \"test-curl-6968-1d5714\" (UID: \"a32621a9-c2a4-4864-b89c-5a761c33a256\") " pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:23.880380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.880297 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k78h\" (UniqueName: \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\") pod \"test-curl-6968-1d5714\" (UID: \"a32621a9-c2a4-4864-b89c-5a761c33a256\") " pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:23.889819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:23.889794 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k78h\" (UniqueName: \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\") pod \"test-curl-6968-1d5714\" (UID: \"a32621a9-c2a4-4864-b89c-5a761c33a256\") " pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:24.027228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:24.027196 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:24.396896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:24.383536 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-1d5714"] Nov 01 22:40:24.838995 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:24.838942 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-ec3f01" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" containerName="test-curl-6968-ec3f01" containerID="cri-o://358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c" gracePeriod=2 Nov 01 22:40:24.840131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:24.839824 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-1d5714" event={"ID":"a32621a9-c2a4-4864-b89c-5a761c33a256","Type":"ContainerStarted","Data":"64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc"} Nov 01 22:40:24.840131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:24.839922 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-1d5714" event={"ID":"a32621a9-c2a4-4864-b89c-5a761c33a256","Type":"ContainerStarted","Data":"c774953fce1ac4e0113ceca9fead5a30f1c35228056296011d07793a96e6ca08"} Nov 01 22:40:25.500541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:25.500477 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-1d5714" podStartSLOduration=2.500456723 podStartE2EDuration="2.500456723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:24.870755128 +0000 UTC m=+994.646741990" watchObservedRunningTime="2026-11-01 22:40:25.500456723 +0000 UTC m=+995.276443555" Nov 01 22:40:25.502467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:25.502287 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-1d5714"] Nov 01 22:40:25.513069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:25.507689 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-1d5714"] Nov 01 22:40:26.881670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:26.874421 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-1d5714" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" containerName="test-curl-6968-1d5714" containerID="cri-o://64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc" gracePeriod=2 Nov 01 22:40:27.289095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.289065 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:27.325024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.324990 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\" (UniqueName: \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\") pod \"16d85602-8b11-416e-bd85-9f582bc86b64\" (UID: \"16d85602-8b11-416e-bd85-9f582bc86b64\") " Nov 01 22:40:27.328207 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.328167 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq" pod "16d85602-8b11-416e-bd85-9f582bc86b64" (UID: "16d85602-8b11-416e-bd85-9f582bc86b64"). InnerVolumeSpecName "kube-api-access-pt2qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:27.426735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.426705 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2qq\" (UniqueName: \"kubernetes.io/projected/16d85602-8b11-416e-bd85-9f582bc86b64-kube-api-access-pt2qq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:27.901434 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.901343 5778 generic.go:356] "Generic (PLEG): container finished" podID="16d85602-8b11-416e-bd85-9f582bc86b64" containerID="358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c" exitCode=137 Nov 01 22:40:27.902207 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.902185 5778 scope.go:122] "RemoveContainer" containerID="358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c" Nov 01 22:40:27.902598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:27.902568 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:28.001878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:28.000096 5778 scope.go:122] "RemoveContainer" containerID="358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c" Nov 01 22:40:28.001878 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:28.001337 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c\": container with ID starting with 358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c not found: ID does not exist" containerID="358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c" Nov 01 22:40:28.001878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:28.001381 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c"} err="failed to get container status \"358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c\": rpc error: code = NotFound desc = could not find container \"358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c\": container with ID starting with 358056d70451910f29c312efd4c9e91369c77c5a86146c521df44eba350de89c not found: ID does not exist" Nov 01 22:40:28.073131 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:28.073089 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ec3f01\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" pod="opendatahub/test-curl-6968-ec3f01" Nov 01 22:40:28.408903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:28.396720 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d85602-8b11-416e-bd85-9f582bc86b64" path="/var/lib/kubelet/pods/16d85602-8b11-416e-bd85-9f582bc86b64/volumes" Nov 01 22:40:29.178613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.178469 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:29.347067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.339377 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\" (UniqueName: \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\") pod \"a32621a9-c2a4-4864-b89c-5a761c33a256\" (UID: \"a32621a9-c2a4-4864-b89c-5a761c33a256\") " Nov 01 22:40:29.395880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.394226 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h" pod "a32621a9-c2a4-4864-b89c-5a761c33a256" (UID: "a32621a9-c2a4-4864-b89c-5a761c33a256"). InnerVolumeSpecName "kube-api-access-5k78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:29.460965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.460546 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k78h\" (UniqueName: \"kubernetes.io/projected/a32621a9-c2a4-4864-b89c-5a761c33a256-kube-api-access-5k78h\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:29.921180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.921143 5778 generic.go:356] "Generic (PLEG): container finished" podID="a32621a9-c2a4-4864-b89c-5a761c33a256" containerID="64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc" exitCode=137 Nov 01 22:40:29.921372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.921220 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:29.921372 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.921235 5778 scope.go:122] "RemoveContainer" containerID="64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc" Nov 01 22:40:29.937248 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:29.937216 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-1d5714\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:29.939289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.939132 5778 scope.go:122] "RemoveContainer" containerID="64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc" Nov 01 22:40:29.939518 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:29.939489 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc\": container with ID starting with 64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc not found: ID does not exist" containerID="64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc" Nov 01 22:40:29.939571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:29.939531 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc"} err="failed to get container status \"64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc\": rpc error: code = NotFound desc = could not find container \"64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc\": container with ID starting with 64d5291ce1aa9ea5b762383d20962da2032544e4674a13f8e1434230379855dc not found: ID does not exist" Nov 01 22:40:30.385822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:30.385731 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" path="/var/lib/kubelet/pods/a32621a9-c2a4-4864-b89c-5a761c33a256/volumes" Nov 01 22:40:30.386924 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:30.386892 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-1d5714\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" pod="opendatahub/test-curl-6968-1d5714" Nov 01 22:40:31.384022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.383978 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:40:31.384802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.384778 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" containerName="test-curl-6968-1d5714" Nov 01 22:40:31.384802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.384802 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" containerName="test-curl-6968-1d5714" Nov 01 22:40:31.385059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.385044 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32621a9-c2a4-4864-b89c-5a761c33a256" containerName="test-curl-6968-1d5714" Nov 01 22:40:31.390360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.390335 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.397016 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.396995 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e61a2d4e-openshift-default-dockercfg-299qs\"" Nov 01 22:40:31.414223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.405844 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:40:31.515371 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515338 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515380 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515515 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515625 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515709 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515806 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.515890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515838 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dff\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.516015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.515979 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.516080 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.516011 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.621970 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622200 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622296 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622335 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622388 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622475 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622624 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622699 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.626875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.622752 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dff\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.648921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.627786 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.648921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.631695 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.648921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.631945 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.648921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.632176 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.649658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.649632 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.655949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.650377 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.655949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.653174 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.655949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.653913 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dff\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.657118 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.657071 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\") pod \"e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:31.744403 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:31.744104 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:32.281753 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.281721 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:40:32.282606 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:40:32.282574 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81530484_5797_4e49_b4e3_c1c3746532bf.slice/crio-bcc84df10ac7722e271af0780a0d8f8697af02cf973ff7e5b40708570319057d WatchSource:0}: Error finding container bcc84df10ac7722e271af0780a0d8f8697af02cf973ff7e5b40708570319057d: Status 404 returned error can't find the container with id bcc84df10ac7722e271af0780a0d8f8697af02cf973ff7e5b40708570319057d Nov 01 22:40:32.285725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.285680 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:40:32.285831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.285776 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:40:32.285831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.285824 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:40:32.998654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.998600 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" event={"ID":"81530484-5797-4e49-b4e3-c1c3746532bf","Type":"ContainerStarted","Data":"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a"} Nov 01 22:40:32.999282 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:32.998699 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" event={"ID":"81530484-5797-4e49-b4e3-c1c3746532bf","Type":"ContainerStarted","Data":"bcc84df10ac7722e271af0780a0d8f8697af02cf973ff7e5b40708570319057d"} Nov 01 22:40:33.072983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:33.072908 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" podStartSLOduration=2.072884217 podStartE2EDuration="2.072884217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:33.038370494 +0000 UTC m=+1002.814357323" watchObservedRunningTime="2026-11-01 22:40:33.072884217 +0000 UTC m=+1002.848871049" Nov 01 22:40:33.745654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:33.745620 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:33.752480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:33.752449 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:34.005919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:34.005811 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:34.007034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:34.007008 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:40:37.431454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.431419 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:40:37.477843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.477812 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:40:37.478044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.477979 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.482581 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.482557 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-88896ec5-serving-cert\"" Nov 01 22:40:37.577807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.577774 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97jt\" (UniqueName: \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.577988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.577825 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.578074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.578061 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.578128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.578104 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.681938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.681837 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.682452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.682425 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.682637 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.682620 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.694936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.682872 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97jt\" (UniqueName: \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.694936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.686585 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.714655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.713786 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.723882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.716512 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97jt\" (UniqueName: \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.723882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.717153 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\") pod \"maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:37.801882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:37.797253 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:38.149051 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:40:38.147664 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea839df_93c4_441d_9470_77fcf79a4855.slice/crio-607ad65cd321c3381d2c078f1757537cbaa5c5786261528bc341e890666bcbd1 WatchSource:0}: Error finding container 607ad65cd321c3381d2c078f1757537cbaa5c5786261528bc341e890666bcbd1: Status 404 returned error can't find the container with id 607ad65cd321c3381d2c078f1757537cbaa5c5786261528bc341e890666bcbd1 Nov 01 22:40:38.155122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:38.155097 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:40:39.063968 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:39.063864 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" event={"ID":"8ea839df-93c4-441d-9470-77fcf79a4855","Type":"ContainerStarted","Data":"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9"} Nov 01 22:40:39.063968 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:39.063928 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" event={"ID":"8ea839df-93c4-441d-9470-77fcf79a4855","Type":"ContainerStarted","Data":"607ad65cd321c3381d2c078f1757537cbaa5c5786261528bc341e890666bcbd1"} Nov 01 22:40:39.064508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:39.064069 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:39.099255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:39.099200 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" podStartSLOduration=1.562306559 podStartE2EDuration="2.0991864s" totalImagesPullingTime="536.879841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:39.096659877 +0000 UTC m=+1008.872646705" watchObservedRunningTime="2026-11-01 22:40:39.0991864 +0000 UTC m=+1008.875173235" Nov 01 22:40:43.746953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.746916 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:40:43.753277 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.753256 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.768674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.768361 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:40:43.772887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.772867 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e61a2d4e-serving-cert\"" Nov 01 22:40:43.786087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.786055 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.786233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.786213 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kntrb\" (UniqueName: \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.786450 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.786424 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.786490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.786477 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.888204 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.888177 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kntrb\" (UniqueName: \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.888340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.888289 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.888340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.888313 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.888460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.888358 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.889015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.888989 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.891116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.891087 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.891226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.891149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:43.897011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:43.896993 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kntrb\" (UniqueName: \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\") pod \"maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:44.073531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:44.073424 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:44.381431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:44.381405 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:40:44.383168 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:40:44.383136 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ff2d0f_efd8_4d8f_8989_0f60f80c0856.slice/crio-61137e859b48a75b412e19b73e6bc219f00d5ad9a692159deb9c22a0b14b242f WatchSource:0}: Error finding container 61137e859b48a75b412e19b73e6bc219f00d5ad9a692159deb9c22a0b14b242f: Status 404 returned error can't find the container with id 61137e859b48a75b412e19b73e6bc219f00d5ad9a692159deb9c22a0b14b242f Nov 01 22:40:45.077213 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:45.077187 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:40:45.188676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:45.188583 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" event={"ID":"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856","Type":"ContainerStarted","Data":"2913322676a778119ab29a5a64dae1b91c781aaec1abdcb19f091a94043038cc"} Nov 01 22:40:45.188676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:45.188653 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:45.188676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:45.188673 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" event={"ID":"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856","Type":"ContainerStarted","Data":"61137e859b48a75b412e19b73e6bc219f00d5ad9a692159deb9c22a0b14b242f"} Nov 01 22:40:45.225358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:45.225296 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" podStartSLOduration=1.7029995169999999 podStartE2EDuration="2.225277594s" totalImagesPullingTime="522.278077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:45.224389626 +0000 UTC m=+1015.000376467" watchObservedRunningTime="2026-11-01 22:40:45.225277594 +0000 UTC m=+1015.001264423" Nov 01 22:40:48.027879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:48.020936 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.110008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:48.109944 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:48.721873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:48.714186 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:40:48.787898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:48.776599 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.029930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.023112 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.029930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.023683 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:40:49.209984 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:49.208919 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:40:49.217899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.215834 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.223884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.223511 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:49.732880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.730376 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.736345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.736308 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.737543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.737520 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:49.785884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.779985 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.785884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.781667 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:40:50.003933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:49.979057 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.077878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:50.031642 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.225439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:50.225395 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.780874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:50.777759 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.780874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:50.777890 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:40:50.782801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:50.782765 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:40:51.268879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.241784 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.726301 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:40:51.686371 5778 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.214s" Nov 01 22:40:51.726301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.686455 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:51.726301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.686502 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:40:51.726301 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.686598 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:40:51.888422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.888321 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:40:52.000131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:51.999911 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.016747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.016710 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:40:52.019774 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.017228 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.173271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.173210 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.257903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.257796 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:40:52.273129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.273060 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:40:52.352436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:40:52.352412 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:41:14.070865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:14.070817 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:41:14.071406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:14.071105 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" containerName="maas-api" containerID="cri-o://d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9" gracePeriod=30 Nov 01 22:41:15.083052 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.083020 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:41:15.084209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.084178 5778 generic.go:356] "Generic (PLEG): container finished" podID="8ea839df-93c4-441d-9470-77fcf79a4855" containerID="d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9" exitCode=0 Nov 01 22:41:15.084436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.084278 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" event={"ID":"8ea839df-93c4-441d-9470-77fcf79a4855","Type":"ContainerDied","Data":"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9"} Nov 01 22:41:15.084520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.084491 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" event={"ID":"8ea839df-93c4-441d-9470-77fcf79a4855","Type":"ContainerDied","Data":"607ad65cd321c3381d2c078f1757537cbaa5c5786261528bc341e890666bcbd1"} Nov 01 22:41:15.084520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.084519 5778 scope.go:122] "RemoveContainer" containerID="d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9" Nov 01 22:41:15.125058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.123781 5778 scope.go:122] "RemoveContainer" containerID="d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9" Nov 01 22:41:15.125058 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:15.124290 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9\": container with ID starting with d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9 not found: ID does not exist" containerID="d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9" Nov 01 22:41:15.125058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.124332 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9"} err="failed to get container status \"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9\": rpc error: code = NotFound desc = could not find container \"d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9\": container with ID starting with d0dbde932dfd24241da3715ca6fdfdee9f02015d5984d2edc26afeb7a81187c9 not found: ID does not exist" Nov 01 22:41:15.179874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.166018 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\" (UniqueName: \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\") pod \"8ea839df-93c4-441d-9470-77fcf79a4855\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " Nov 01 22:41:15.179874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.166126 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\") pod \"8ea839df-93c4-441d-9470-77fcf79a4855\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " Nov 01 22:41:15.179874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.166173 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\") pod \"8ea839df-93c4-441d-9470-77fcf79a4855\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " Nov 01 22:41:15.179874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.166266 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\") pod \"8ea839df-93c4-441d-9470-77fcf79a4855\" (UID: \"8ea839df-93c4-441d-9470-77fcf79a4855\") " Nov 01 22:41:15.179874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.174767 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca" pod "8ea839df-93c4-441d-9470-77fcf79a4855" (UID: "8ea839df-93c4-441d-9470-77fcf79a4855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:15.198105 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.195137 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt" pod "8ea839df-93c4-441d-9470-77fcf79a4855" (UID: "8ea839df-93c4-441d-9470-77fcf79a4855"). InnerVolumeSpecName "kube-api-access-l97jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:15.198105 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.195176 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs" pod "8ea839df-93c4-441d-9470-77fcf79a4855" (UID: "8ea839df-93c4-441d-9470-77fcf79a4855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:15.226093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.218178 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls" pod "8ea839df-93c4-441d-9470-77fcf79a4855" (UID: "8ea839df-93c4-441d-9470-77fcf79a4855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:15.268787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.268751 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97jt\" (UniqueName: \"kubernetes.io/projected/8ea839df-93c4-441d-9470-77fcf79a4855-kube-api-access-l97jt\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:15.269041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.269024 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea839df-93c4-441d-9470-77fcf79a4855-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:15.269140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.269128 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:15.269228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:15.269217 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea839df-93c4-441d-9470-77fcf79a4855-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:16.103898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.103190 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" Nov 01 22:41:16.147750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.147712 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:41:16.152280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.152253 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr"] Nov 01 22:41:16.187168 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.187135 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:41:16.188159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.188136 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" containerName="maas-api" Nov 01 22:41:16.188159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.188159 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" containerName="maas-api" Nov 01 22:41:16.192435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.188411 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" containerName="maas-api" Nov 01 22:41:16.193631 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.193611 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.209614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.209574 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:41:16.210272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.210250 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b8b59d8d-gw-openshift-default-dockercfg-rn6gr\"" Nov 01 22:41:16.302113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302079 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302242 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7pf\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302265 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302299 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302319 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302468 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302491 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302596 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.302698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.302683 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.376611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.376534 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" path="/var/lib/kubelet/pods/8ea839df-93c4-441d-9470-77fcf79a4855/volumes" Nov 01 22:41:16.403717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.403682 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.403717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.403715 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.403772 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.403792 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.403923 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404037 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404073 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404140 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404256 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7pf\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404288 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404335 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404684 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.404932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404909 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.405012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.404943 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.406354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.406327 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.407069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.407050 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.415544 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.415516 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7pf\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.417172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.417153 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\") pod \"e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:16.539882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:16.536141 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:17.319827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:17.319783 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:41:17.474556 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:41:17.474516 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2a4927_043c_422d_a50d_fa43a8c65676.slice/crio-02a13fd60bc77ff9c0ef523978849928464ad073b7d6cb0c05b064591d523453 WatchSource:0}: Error finding container 02a13fd60bc77ff9c0ef523978849928464ad073b7d6cb0c05b064591d523453: Status 404 returned error can't find the container with id 02a13fd60bc77ff9c0ef523978849928464ad073b7d6cb0c05b064591d523453 Nov 01 22:41:17.557661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:17.557614 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:17.557984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:17.557955 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:17.558187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:17.558161 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:17.988878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:17.979954 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.082919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:18.073381 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88896ec5-9998cf5bc-v4vcr" podUID="8ea839df-93c4-441d-9470-77fcf79a4855" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.88:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.448389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:18.448278 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" event={"ID":"cd2a4927-043c-422d-a50d-fa43a8c65676","Type":"ContainerStarted","Data":"02a13fd60bc77ff9c0ef523978849928464ad073b7d6cb0c05b064591d523453"} Nov 01 22:41:18.792091 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:18.790948 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.802887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:18.801938 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.010583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.006044 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:19.010583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.006346 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:41:19.010583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.006911 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:41:19.123289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.122661 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.290404 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:19.267499 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:41:19.544880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.536603 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" event={"ID":"cd2a4927-043c-422d-a50d-fa43a8c65676","Type":"ContainerStarted","Data":"c59d224351c19302914c53c63a2a09e9a5f2cde5d351963c9c0d20704a743ab8"} Nov 01 22:41:19.551156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.551127 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:19.568387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.568350 5778 patch_prober.go:29] interesting pod/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:41:19.589064 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.572118 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 01 22:41:19.638731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.638668 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" podStartSLOduration=3.620486035 podStartE2EDuration="3.620486035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:19.590032656 +0000 UTC m=+1049.366019487" watchObservedRunningTime="2026-11-01 22:41:19.620486035 +0000 UTC m=+1049.396472864" Nov 01 22:41:19.795170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.795073 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.796006 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:19.795981 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:41:20.112931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:20.108989 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:20.186911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:20.174731 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.186911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:20.175268 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:41:21.016878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:20.997553 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.139940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:21.029954 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:21.139940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:21.039758 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.139940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:21.118399 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.181052 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:21.181006 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.184570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:21.184543 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:22.058223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.058192 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:22.061109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.061083 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:41:22.139877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.130946 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.139877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.131319 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:22.139877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.131575 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:41:22.139877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.132204 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:41:22.205910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.199575 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.433877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.432450 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:22.433877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.432537 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.850761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:22.850602 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 01 22:41:23.026006 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.025955 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.118919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.118811 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:23.127140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.127101 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.142140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.135287 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.146917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.135322 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:23.207937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.207895 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.650003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.649961 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:41:23.702976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:23.702933 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.127959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.127920 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:41:24.142623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.142581 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.246886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.207985 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.249469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.234100 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:41:24.249469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.234162 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.292887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.292190 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b07f40e8-openshift-default-dockercfg-cxmq5\"" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.380460 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.380526 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.380601 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.380835 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.380978 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2fk\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.381148 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.381181 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.381241 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.397961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.381269 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.482886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.482827 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.483186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.483158 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.483354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.483337 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.483684 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.483666 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.483822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.483808 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2fk\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.484082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.484059 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.484209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.484194 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.484346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.484332 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.484445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.484431 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.485064 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.485041 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.499201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.485812 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.499201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.493199 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.499201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.493448 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.499201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.493671 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.597779 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.597705 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.599579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.599549 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.661022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.614692 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2fk\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.661022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.646236 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\") pod \"e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.694182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.694151 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:24.703383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.703338 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.029552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:24.848403 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:41:25.223628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.156656 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:41:25.223628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.171200 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:25.256523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.256457 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.479519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.479492 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:41:25.497353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.483109 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:41:25.975982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.975950 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:25.976239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.976214 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.983345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.983313 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:25.999929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:25.983579 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:26.035201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.030007 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:41:26.119287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.119247 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:26.121232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.121208 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:41:26.191622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.182438 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:26.191622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.182533 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:26.191622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.182578 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:41:26.222882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.220644 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:41:26.343361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.343323 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:41:26.881196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.881164 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" event={"ID":"3dfee347-a686-429b-8e2d-62e841ab8d9a","Type":"ContainerStarted","Data":"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498"} Nov 01 22:41:26.881464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.881447 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" event={"ID":"3dfee347-a686-429b-8e2d-62e841ab8d9a","Type":"ContainerStarted","Data":"3ca12e8e6b23f21db4d4467bf7b7664d65b78d81ee2a7fe710e23f3b20d2bc54"} Nov 01 22:41:26.905325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:26.905266 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" podStartSLOduration=3.905246675 podStartE2EDuration="3.905246675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:26.903051542 +0000 UTC m=+1056.679038373" watchObservedRunningTime="2026-11-01 22:41:26.905246675 +0000 UTC m=+1056.681233506" Nov 01 22:41:27.262882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:27.240559 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:41:27.274950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:27.274910 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" podUID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" containerName="maas-api" containerID="cri-o://2913322676a778119ab29a5a64dae1b91c781aaec1abdcb19f091a94043038cc" gracePeriod=30 Nov 01 22:41:27.706638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:27.704498 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:27.711159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:27.711124 5778 patch_prober.go:29] interesting pod/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 22:41:27.711374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:27.711345 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 22:41:28.005624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.005588 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:41:28.006106 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.006076 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" containerName="istio-proxy" containerID="cri-o://9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a" gracePeriod=30 Nov 01 22:41:28.307879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.213835 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:28.307879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.267137 5778 generic.go:356] "Generic (PLEG): container finished" podID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" containerID="2913322676a778119ab29a5a64dae1b91c781aaec1abdcb19f091a94043038cc" exitCode=0 Nov 01 22:41:28.307879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.268299 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" event={"ID":"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856","Type":"ContainerDied","Data":"2913322676a778119ab29a5a64dae1b91c781aaec1abdcb19f091a94043038cc"} Nov 01 22:41:28.511296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.509804 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:41:28.562992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.561813 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\" (UniqueName: \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\") pod \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " Nov 01 22:41:28.563794 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.563765 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\") pod \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " Nov 01 22:41:28.564067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.564047 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\") pod \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " Nov 01 22:41:28.564377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.564361 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\") pod \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\" (UID: \"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856\") " Nov 01 22:41:28.566004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.565985 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb" pod "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" (UID: "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856"). InnerVolumeSpecName "kube-api-access-kntrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:28.566602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.566581 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca" pod "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" (UID: "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:28.569438 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.569414 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs" pod "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" (UID: "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:28.569734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.569716 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls" pod "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" (UID: "a2ff2d0f-efd8-4d8f-8989-0f60f80c0856"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:28.678887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.669356 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:28.678887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.669395 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:28.678887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.669410 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:28.678887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.669423 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kntrb\" (UniqueName: \"kubernetes.io/projected/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856-kube-api-access-kntrb\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:28.714680 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:28.713026 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:29.424876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.423882 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" Nov 01 22:41:29.429876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.428949 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx" event={"ID":"a2ff2d0f-efd8-4d8f-8989-0f60f80c0856","Type":"ContainerDied","Data":"61137e859b48a75b412e19b73e6bc219f00d5ad9a692159deb9c22a0b14b242f"} Nov 01 22:41:29.429876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.429038 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:29.429876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.429083 5778 scope.go:122] "RemoveContainer" containerID="2913322676a778119ab29a5a64dae1b91c781aaec1abdcb19f091a94043038cc" Nov 01 22:41:29.437056 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.437011 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:41:29.572100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.572065 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:41:29.589537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:29.589506 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e61a2d4e-c5dfb48f6-thdtx"] Nov 01 22:41:30.385303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:30.385266 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" path="/var/lib/kubelet/pods/a2ff2d0f-efd8-4d8f-8989-0f60f80c0856/volumes" Nov 01 22:41:33.013640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.013599 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:41:33.014696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.014669 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" containerName="maas-api" Nov 01 22:41:33.014696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.014695 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" containerName="maas-api" Nov 01 22:41:33.015934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.014914 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff2d0f-efd8-4d8f-8989-0f60f80c0856" containerName="maas-api" Nov 01 22:41:33.051373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.051340 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:41:33.051705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.051680 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.066431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.065414 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b8b59d8d-serving-cert\"" Nov 01 22:41:33.115217 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.115186 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxz6l\" (UniqueName: \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.115367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.115257 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.115435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.115388 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.115523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.115500 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.216944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.216907 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.217220 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:33.217072 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-serving-cert: secret "maas-api-e2e-migrate-b8b59d8d-serving-cert" not found Nov 01 22:41:33.217220 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.217159 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.217421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.217392 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxz6l\" (UniqueName: \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.217488 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:33.217452 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls podName:3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6 nodeName:}" failed. No retries permitted until 2026-11-01 22:41:33.717427448 +0000 UTC m=+1063.493414255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls") pod "maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" (UID: "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6") : secret "maas-api-e2e-migrate-b8b59d8d-serving-cert" not found Nov 01 22:41:33.217561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.217542 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.217820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.217802 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.223894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.222259 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.229677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.229652 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxz6l\" (UniqueName: \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.735961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.735918 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.777562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.777488 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:33.982072 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:33.982021 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:34.344811 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:34.344784 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:41:34.345519 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:41:34.345495 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf2f829_0f6f_459f_b2f6_dda28a1fa0f6.slice/crio-90724566fc3b0a6192757e1cb1f0c30cc17c404d829f18e90ba3bf70b36118aa WatchSource:0}: Error finding container 90724566fc3b0a6192757e1cb1f0c30cc17c404d829f18e90ba3bf70b36118aa: Status 404 returned error can't find the container with id 90724566fc3b0a6192757e1cb1f0c30cc17c404d829f18e90ba3bf70b36118aa Nov 01 22:41:34.485536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:34.485503 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" event={"ID":"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6","Type":"ContainerStarted","Data":"90724566fc3b0a6192757e1cb1f0c30cc17c404d829f18e90ba3bf70b36118aa"} Nov 01 22:41:34.611143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:34.609475 5778 patch_prober.go:29] interesting pod/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": read tcp 10.132.0.2:59510->10.132.0.87:15021: read: connection reset by peer" start-of-body= Nov 01 22:41:34.611143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:34.609550 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": read tcp 10.132.0.2:59510->10.132.0.87:15021: read: connection reset by peer" Nov 01 22:41:34.951174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:34.951146 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:41:35.080067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080034 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080126 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080202 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080254 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080273 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080350 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080386 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080512 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080442 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:35.080582 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080527 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:35.080582 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080564 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:35.080697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080600 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080645 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\") pod \"81530484-5797-4e49-b4e3-c1c3746532bf\" (UID: \"81530484-5797-4e49-b4e3-c1c3746532bf\") " Nov 01 22:41:35.080814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080790 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:35.080958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.080936 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:35.081394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.081301 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.081394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.081328 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.081394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.081344 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.081394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.081358 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.081394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.081372 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81530484-5797-4e49-b4e3-c1c3746532bf-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.083420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.083397 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:35.083523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.083421 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:35.083523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.083423 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "kube-api-access-x8dff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:35.083523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.083420 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo" pod "81530484-5797-4e49-b4e3-c1c3746532bf" (UID: "81530484-5797-4e49-b4e3-c1c3746532bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:41:35.182956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.182928 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81530484-5797-4e49-b4e3-c1c3746532bf-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.182956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.182956 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.183184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.182966 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dff\" (UniqueName: \"kubernetes.io/projected/81530484-5797-4e49-b4e3-c1c3746532bf-kube-api-access-x8dff\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.183184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.182975 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81530484-5797-4e49-b4e3-c1c3746532bf-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:35.494246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.494160 5778 generic.go:356] "Generic (PLEG): container finished" podID="81530484-5797-4e49-b4e3-c1c3746532bf" containerID="9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a" exitCode=0 Nov 01 22:41:35.494246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.494203 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" Nov 01 22:41:35.494733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.494251 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" event={"ID":"81530484-5797-4e49-b4e3-c1c3746532bf","Type":"ContainerDied","Data":"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a"} Nov 01 22:41:35.494733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.494318 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c" event={"ID":"81530484-5797-4e49-b4e3-c1c3746532bf","Type":"ContainerDied","Data":"bcc84df10ac7722e271af0780a0d8f8697af02cf973ff7e5b40708570319057d"} Nov 01 22:41:35.494733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.494351 5778 scope.go:122] "RemoveContainer" containerID="9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a" Nov 01 22:41:35.496508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.496479 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" event={"ID":"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6","Type":"ContainerStarted","Data":"c2956087d6609fe7e3002fb74b7dfc98dc2fabf6a040f499540db3d957099763"} Nov 01 22:41:35.496662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.496551 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:35.508117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.507058 5778 scope.go:122] "RemoveContainer" containerID="9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a" Nov 01 22:41:35.508117 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:35.507845 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a\": container with ID starting with 9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a not found: ID does not exist" containerID="9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a" Nov 01 22:41:35.508117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.507895 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a"} err="failed to get container status \"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a\": rpc error: code = NotFound desc = could not find container \"9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a\": container with ID starting with 9e6d3c42b6767cd1832edf58926326f72b0f6d9b7357523d73888185a3c78d1a not found: ID does not exist" Nov 01 22:41:35.516044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.515996 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" podStartSLOduration=3.080149309 podStartE2EDuration="3.51598359s" totalImagesPullingTime="435.834281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:35.5149841 +0000 UTC m=+1065.290970964" watchObservedRunningTime="2026-11-01 22:41:35.51598359 +0000 UTC m=+1065.291970419" Nov 01 22:41:35.535912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.535883 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:41:35.541698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:35.541677 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e61a2d4e-openshift-default-79db9d568f-k6z5c"] Nov 01 22:41:36.329637 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.329607 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:41:36.330304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.330289 5778 cpu_manager.go:414] "containerMap: removing container" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" containerName="istio-proxy" Nov 01 22:41:36.330361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.330308 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" containerName="istio-proxy" Nov 01 22:41:36.330432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.330424 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" containerName="istio-proxy" Nov 01 22:41:36.335441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.335421 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.342477 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.342457 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b07f40e8-serving-cert\"" Nov 01 22:41:36.348599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.348578 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:41:36.395891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.395805 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81530484-5797-4e49-b4e3-c1c3746532bf" path="/var/lib/kubelet/pods/81530484-5797-4e49-b4e3-c1c3746532bf/volumes" Nov 01 22:41:36.506015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.505984 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s4t\" (UniqueName: \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.506015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.506015 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.506502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.506047 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.506502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.506273 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.608924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.608007 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.608924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.608270 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s4t\" (UniqueName: \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.608924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.608300 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.608924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.608415 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.614838 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.609360 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.614838 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.613903 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.631109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.631068 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s4t\" (UniqueName: \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.645125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.641552 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\") pod \"maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:36.664491 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:36.663259 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:37.133877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:37.122373 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:41:37.513428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:37.513399 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" event={"ID":"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2","Type":"ContainerStarted","Data":"835b81f1b52100510277df57972dbd995ca6d04dbdff4e2a0faa1a5e30dd4cc4"} Nov 01 22:41:38.523697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:38.523659 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" event={"ID":"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2","Type":"ContainerStarted","Data":"cdb745606499dd6a773ceee1c8d28640b18142bd5627083fa483300dd80e472c"} Nov 01 22:41:38.524127 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:38.524051 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:38.545687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:38.545642 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" podStartSLOduration=2.034737122 podStartE2EDuration="2.545629974s" totalImagesPullingTime="510.892852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:38.545610591 +0000 UTC m=+1068.321597443" watchObservedRunningTime="2026-11-01 22:41:38.545629974 +0000 UTC m=+1068.321616803" Nov 01 22:41:41.510083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:41.510055 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:41:44.540782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:41:44.540750 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:41:48.773673 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:41:48.773634 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:42:03.205313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.205222 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:42:03.205758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.205619 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" podUID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" containerName="maas-api" containerID="cri-o://c2956087d6609fe7e3002fb74b7dfc98dc2fabf6a040f499540db3d957099763" gracePeriod=30 Nov 01 22:42:03.756798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.756763 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:42:03.757263 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.757236 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" containerID="cri-o://c59d224351c19302914c53c63a2a09e9a5f2cde5d351963c9c0d20704a743ab8" gracePeriod=30 Nov 01 22:42:03.825757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.814541 5778 generic.go:356] "Generic (PLEG): container finished" podID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" containerID="c2956087d6609fe7e3002fb74b7dfc98dc2fabf6a040f499540db3d957099763" exitCode=0 Nov 01 22:42:03.825757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:03.814597 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" event={"ID":"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6","Type":"ContainerDied","Data":"c2956087d6609fe7e3002fb74b7dfc98dc2fabf6a040f499540db3d957099763"} Nov 01 22:42:04.015555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.015526 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:42:04.068875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.068815 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\") pod \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " Nov 01 22:42:04.069196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.069171 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\" (UniqueName: \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\") pod \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " Nov 01 22:42:04.069360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.069346 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") pod \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " Nov 01 22:42:04.069723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.069707 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\") pod \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\" (UID: \"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6\") " Nov 01 22:42:04.070180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.070152 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca" pod "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" (UID: "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:04.070816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.070795 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:04.071446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.071423 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs" pod "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" (UID: "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:04.072377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.072355 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l" pod "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" (UID: "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6"). InnerVolumeSpecName "kube-api-access-xxz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:04.078178 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.078143 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls" pod "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" (UID: "3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:04.173297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.173206 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:04.173297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.173245 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxz6l\" (UniqueName: \"kubernetes.io/projected/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-kube-api-access-xxz6l\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:04.173297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.173260 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:04.825843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.825812 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" Nov 01 22:42:04.826389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.825842 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2" event={"ID":"3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6","Type":"ContainerDied","Data":"90724566fc3b0a6192757e1cb1f0c30cc17c404d829f18e90ba3bf70b36118aa"} Nov 01 22:42:04.826389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.825944 5778 scope.go:122] "RemoveContainer" containerID="c2956087d6609fe7e3002fb74b7dfc98dc2fabf6a040f499540db3d957099763" Nov 01 22:42:04.868907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.867314 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:42:04.880268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.880238 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b8b59d8d-d5d8947b4-lcjl2"] Nov 01 22:42:04.904079 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.904054 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 22:42:04.904716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.904702 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" containerName="maas-api" Nov 01 22:42:04.904767 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.904717 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" containerName="maas-api" Nov 01 22:42:04.904893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.904877 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" containerName="maas-api" Nov 01 22:42:04.909787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.909742 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.930069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.924049 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-7366bf-openshift-default-dockercfg-9nnps\"" Nov 01 22:42:04.930165 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.930081 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996216 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996427 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996520 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996656 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqpm\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996683 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996724 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:04.998642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996811 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.001063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:04.996931 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.103934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.103829 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.104674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.104650 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.105361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.105343 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.109465 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.109443 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.109641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.109620 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.110085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.110062 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.118826 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.118795 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqpm\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.118975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.118842 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.118975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.118903 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.126879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.121640 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.126879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.109292 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.126879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.104558 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.126879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.122243 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.126879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.110587 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.258918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.181199 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqpm\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.297946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.288813 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.349867 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.349808 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.357269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.357156 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:05.533150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:05.533118 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:06.028407 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.028378 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 22:42:06.069932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.063227 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:06.069932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.063320 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:06.069932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.063363 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:06.417109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.414794 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6" path="/var/lib/kubelet/pods/3cf2f829-0f6f-459f-b2f6-dda28a1fa0f6/volumes" Nov 01 22:42:06.869999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.869916 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerStarted","Data":"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6"} Nov 01 22:42:06.869999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.869978 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerStarted","Data":"3fd9a7325f8589a7a55883b2783b23e340fc92f110f05d2f5497eab35d7da7b9"} Nov 01 22:42:06.901322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:06.901243 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podStartSLOduration=2.901196077 podStartE2EDuration="2.901196077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:06.895969572 +0000 UTC m=+1096.671956401" watchObservedRunningTime="2026-11-01 22:42:06.901196077 +0000 UTC m=+1096.677182905" Nov 01 22:42:07.068289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.068091 5778 patch_prober.go:29] interesting pod/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:42:07.068289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.068167 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:42:07.236899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.235156 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:42:07.236899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.235483 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" podUID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" containerName="maas-api" containerID="cri-o://cdb745606499dd6a773ceee1c8d28640b18142bd5627083fa483300dd80e472c" gracePeriod=30 Nov 01 22:42:07.572879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.536830 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:07.623221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.623170 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:07.897305 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.897276 5778 generic.go:356] "Generic (PLEG): container finished" podID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" containerID="cdb745606499dd6a773ceee1c8d28640b18142bd5627083fa483300dd80e472c" exitCode=0 Nov 01 22:42:07.898352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.898331 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" event={"ID":"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2","Type":"ContainerDied","Data":"cdb745606499dd6a773ceee1c8d28640b18142bd5627083fa483300dd80e472c"} Nov 01 22:42:07.898420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.898383 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:07.899464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.899447 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:07.925729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.925708 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:42:07.978892 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.978844 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\" (UniqueName: \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\") pod \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " Nov 01 22:42:07.979076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.979054 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\") pod \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " Nov 01 22:42:07.979148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.979085 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\") pod \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " Nov 01 22:42:07.979208 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.979179 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\") pod \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\" (UID: \"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2\") " Nov 01 22:42:07.980076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.979612 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca" pod "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" (UID: "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:07.980076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.980045 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:07.982545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.982512 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs" pod "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" (UID: "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:07.982701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.982676 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t" pod "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" (UID: "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2"). InnerVolumeSpecName "kube-api-access-m7s4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:07.982975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:07.982954 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls" pod "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" (UID: "5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:08.083611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.083518 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:08.083611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.083560 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:08.083611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.083575 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7s4t\" (UniqueName: \"kubernetes.io/projected/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2-kube-api-access-m7s4t\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:08.253693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.253657 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:42:08.254408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.254375 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerName="istio-proxy" containerID="cri-o://fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498" gracePeriod=30 Nov 01 22:42:08.907001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.906972 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" Nov 01 22:42:08.907001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.906998 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9" event={"ID":"5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2","Type":"ContainerDied","Data":"835b81f1b52100510277df57972dbd995ca6d04dbdff4e2a0faa1a5e30dd4cc4"} Nov 01 22:42:08.907208 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.907066 5778 scope.go:122] "RemoveContainer" containerID="cdb745606499dd6a773ceee1c8d28640b18142bd5627083fa483300dd80e472c" Nov 01 22:42:08.909048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.909020 5778 generic.go:356] "Generic (PLEG): container finished" podID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerID="c59d224351c19302914c53c63a2a09e9a5f2cde5d351963c9c0d20704a743ab8" exitCode=0 Nov 01 22:42:08.909263 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.909219 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" event={"ID":"cd2a4927-043c-422d-a50d-fa43a8c65676","Type":"ContainerDied","Data":"c59d224351c19302914c53c63a2a09e9a5f2cde5d351963c9c0d20704a743ab8"} Nov 01 22:42:08.933939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.933908 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:42:08.939924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:08.939901 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b07f40e8-5bbd47bc87-qhqt9"] Nov 01 22:42:09.066720 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.066697 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:42:09.209053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209025 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209057 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209137 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209229 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209252 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209274 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209290 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209348 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209379 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209504 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\") pod \"cd2a4927-043c-422d-a50d-fa43a8c65676\" (UID: \"cd2a4927-043c-422d-a50d-fa43a8c65676\") " Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209518 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:09.209542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209536 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:09.210068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209553 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:09.210068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.209737 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:09.210408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.210337 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.210458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.210414 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.210458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.210430 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2a4927-043c-422d-a50d-fa43a8c65676-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.210458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.210444 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.210553 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.210458 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.211597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.211577 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:09.211906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.211888 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:09.212176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.212158 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "kube-api-access-sg7pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:09.212233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.212183 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token" pod "cd2a4927-043c-422d-a50d-fa43a8c65676" (UID: "cd2a4927-043c-422d-a50d-fa43a8c65676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:09.312308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.312151 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.312308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.312188 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.312308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.312205 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.312308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.312219 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7pf\" (UniqueName: \"kubernetes.io/projected/cd2a4927-043c-422d-a50d-fa43a8c65676-kube-api-access-sg7pf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:09.895869 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.895818 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 22:42:09.897062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.896759 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" containerName="maas-api" Nov 01 22:42:09.897228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.897214 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" containerName="maas-api" Nov 01 22:42:09.897346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.897337 5778 cpu_manager.go:414] "containerMap: removing container" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" Nov 01 22:42:09.897414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.897406 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" Nov 01 22:42:09.898308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.897722 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" containerName="maas-api" Nov 01 22:42:09.898445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.898430 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" containerName="istio-proxy" Nov 01 22:42:09.926911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.908775 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:09.930114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.930081 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 22:42:09.939187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.939150 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" event={"ID":"cd2a4927-043c-422d-a50d-fa43a8c65676","Type":"ContainerDied","Data":"02a13fd60bc77ff9c0ef523978849928464ad073b7d6cb0c05b064591d523453"} Nov 01 22:42:09.939338 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.939232 5778 scope.go:122] "RemoveContainer" containerID="c59d224351c19302914c53c63a2a09e9a5f2cde5d351963c9c0d20704a743ab8" Nov 01 22:42:09.939338 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:09.939271 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk" Nov 01 22:42:10.000889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.000413 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ee07ad-openshift-default-dockercfg-krtk4\"" Nov 01 22:42:10.039402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.037881 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:42:10.042111 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.042084 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b8b59d8d-gw-openshift-default-54fcc48b89-hlcxk"] Nov 01 22:42:10.043003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.042948 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043037 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whc48\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043086 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043128 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043217 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043250 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043327 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043360 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.043669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.043438 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145136 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145187 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145248 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145427 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145499 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whc48\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145556 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145601 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145691 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145722 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.147893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.145738 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.153184 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.156205 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.158904 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.159413 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.162044 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.162052 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whc48\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.165461 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.178894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.167981 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.237166 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.237134 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:10.433757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.433719 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2" path="/var/lib/kubelet/pods/5f71eb5e-0079-40e4-a18e-b5dc1ce8adb2/volumes" Nov 01 22:42:10.435242 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.435212 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2a4927-043c-422d-a50d-fa43a8c65676" path="/var/lib/kubelet/pods/cd2a4927-043c-422d-a50d-fa43a8c65676/volumes" Nov 01 22:42:10.551310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.551281 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 22:42:10.551938 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:42:10.551908 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98c93a2_a15f_4e57_bd94_e5b685a879d4.slice/crio-99d8c55a8cdc39a7d09fa309367c8dd0d96c39821eef739994e97f6c8585e022 WatchSource:0}: Error finding container 99d8c55a8cdc39a7d09fa309367c8dd0d96c39821eef739994e97f6c8585e022: Status 404 returned error can't find the container with id 99d8c55a8cdc39a7d09fa309367c8dd0d96c39821eef739994e97f6c8585e022 Nov 01 22:42:10.555119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.555077 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:10.555222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.555163 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:10.555222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:10.555205 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:11.035549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.035508 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerStarted","Data":"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3"} Nov 01 22:42:11.044917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.039974 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerStarted","Data":"99d8c55a8cdc39a7d09fa309367c8dd0d96c39821eef739994e97f6c8585e022"} Nov 01 22:42:11.154585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.138428 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podStartSLOduration=2.138407055 podStartE2EDuration="2.138407055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:11.119414416 +0000 UTC m=+1100.895401269" watchObservedRunningTime="2026-11-01 22:42:11.138407055 +0000 UTC m=+1100.914393888" Nov 01 22:42:11.243877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.243320 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:11.247876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.244639 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.247876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:11.244695 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:42:12.279155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:12.278988 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:13.078317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.078281 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:13.079291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.079259 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:42:13.586716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.586679 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.662504 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.662654 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.662713 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663492 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663610 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663713 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663785 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663834 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.663917 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\") pod \"3dfee347-a686-429b-8e2d-62e841ab8d9a\" (UID: \"3dfee347-a686-429b-8e2d-62e841ab8d9a\") " Nov 01 22:42:13.665755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.665374 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:13.668156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.668113 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:13.668347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.668183 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:13.668509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.668491 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:13.668655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.668617 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:13.696908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.694149 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:13.708251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.708188 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:13.718726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.718688 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "kube-api-access-8f2fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:13.718869 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.718744 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token" pod "3dfee347-a686-429b-8e2d-62e841ab8d9a" (UID: "3dfee347-a686-429b-8e2d-62e841ab8d9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:13.765840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765809 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.765840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765836 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.765840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765846 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfee347-a686-429b-8e2d-62e841ab8d9a-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765885 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765894 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765904 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfee347-a686-429b-8e2d-62e841ab8d9a-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765912 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765920 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:13.766067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:13.765932 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2fk\" (UniqueName: \"kubernetes.io/projected/3dfee347-a686-429b-8e2d-62e841ab8d9a-kube-api-access-8f2fk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:14.043707 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.043629 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg"] Nov 01 22:42:14.056248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.056223 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be62e240-vj7cg"] Nov 01 22:42:14.086663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.086637 5778 generic.go:356] "Generic (PLEG): container finished" podID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerID="fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498" exitCode=0 Nov 01 22:42:14.086829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.086690 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" Nov 01 22:42:14.086829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.086728 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" event={"ID":"3dfee347-a686-429b-8e2d-62e841ab8d9a","Type":"ContainerDied","Data":"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498"} Nov 01 22:42:14.086829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.086808 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d" event={"ID":"3dfee347-a686-429b-8e2d-62e841ab8d9a","Type":"ContainerDied","Data":"3ca12e8e6b23f21db4d4467bf7b7664d65b78d81ee2a7fe710e23f3b20d2bc54"} Nov 01 22:42:14.087000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.086834 5778 scope.go:122] "RemoveContainer" containerID="fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498" Nov 01 22:42:14.099508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.099493 5778 scope.go:122] "RemoveContainer" containerID="fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498" Nov 01 22:42:14.099768 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:42:14.099749 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498\": container with ID starting with fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498 not found: ID does not exist" containerID="fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498" Nov 01 22:42:14.099820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.099777 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498"} err="failed to get container status \"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498\": rpc error: code = NotFound desc = could not find container \"fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498\": container with ID starting with fe6291444ce5ae50235382cae353d4636b71682639938e89156b9eff7a787498 not found: ID does not exist" Nov 01 22:42:14.120012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.119984 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:42:14.123641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.123619 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b07f40e8-openshift-default-6d599dbb65-hm66d"] Nov 01 22:42:14.376588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.376511 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" path="/var/lib/kubelet/pods/3dfee347-a686-429b-8e2d-62e841ab8d9a/volumes" Nov 01 22:42:14.377200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:14.377180 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29f1133-7b63-46a9-b3bd-4ff7044be449" path="/var/lib/kubelet/pods/e29f1133-7b63-46a9-b3bd-4ff7044be449/volumes" Nov 01 22:42:18.128972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.126058 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 22:42:18.128972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.127125 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerName="istio-proxy" Nov 01 22:42:18.128972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.127147 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerName="istio-proxy" Nov 01 22:42:18.128972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.127356 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfee347-a686-429b-8e2d-62e841ab8d9a" containerName="istio-proxy" Nov 01 22:42:18.135989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.135081 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.140095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.140068 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 22:42:18.143414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.141687 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-7366bf-serving-cert\"" Nov 01 22:42:18.196543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.196508 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vpr\" (UniqueName: \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.196663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.196642 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.196719 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.196697 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.196758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.196721 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.298782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.298751 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vpr\" (UniqueName: \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.299008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.298862 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.299008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.298895 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.299008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.298915 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.299181 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:42:18.299041 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-serving-cert: secret "maas-api-e2e-shared-a-w5-7366bf-serving-cert" not found Nov 01 22:42:18.299181 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:42:18.299133 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls podName:9eae91fb-897c-4288-ac57-86173fe4e051 nodeName:}" failed. No retries permitted until 2026-11-01 22:42:18.799113292 +0000 UTC m=+1108.575100112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls") pod "maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" (UID: "9eae91fb-897c-4288-ac57-86173fe4e051") : secret "maas-api-e2e-shared-a-w5-7366bf-serving-cert" not found Nov 01 22:42:18.299802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.299722 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.302112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.302086 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.310438 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.310415 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vpr\" (UniqueName: \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.811877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.811558 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.824564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:18.824535 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:18.916489 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:42:18.916445 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:42:19.058628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:19.058590 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:19.410720 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:19.410675 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 22:42:19.411179 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:42:19.411070 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eae91fb_897c_4288_ac57_86173fe4e051.slice/crio-43deae3d2ef781931bb0422bdcbf14f2cea876f26944b3140384de95dde98b43 WatchSource:0}: Error finding container 43deae3d2ef781931bb0422bdcbf14f2cea876f26944b3140384de95dde98b43: Status 404 returned error can't find the container with id 43deae3d2ef781931bb0422bdcbf14f2cea876f26944b3140384de95dde98b43 Nov 01 22:42:20.308950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:20.308906 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" event={"ID":"9eae91fb-897c-4288-ac57-86173fe4e051","Type":"ContainerStarted","Data":"dd13864b69cf8aff6475301f9c9401313ec3a3ef0974722bd6715e0f5d95646c"} Nov 01 22:42:20.309125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:20.308962 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" event={"ID":"9eae91fb-897c-4288-ac57-86173fe4e051","Type":"ContainerStarted","Data":"43deae3d2ef781931bb0422bdcbf14f2cea876f26944b3140384de95dde98b43"} Nov 01 22:42:20.309125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:20.308989 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:20.341180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:20.341118 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" podStartSLOduration=1.838564799 podStartE2EDuration="2.341097202s" totalImagesPullingTime="502.532403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:20.331572092 +0000 UTC m=+1110.107558921" watchObservedRunningTime="2026-11-01 22:42:20.341097202 +0000 UTC m=+1110.117084034" Nov 01 22:42:21.466055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.466018 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 22:42:21.472736 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.472705 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.489490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.489460 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ee07ad-serving-cert\"" Nov 01 22:42:21.500004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.499976 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 22:42:21.623222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.623185 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsknq\" (UniqueName: \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.624115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.624083 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.624550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.624526 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.624800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.624780 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.726428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.726344 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.726603 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.726463 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.726603 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.726530 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsknq\" (UniqueName: \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.726731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.726659 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.727307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.727260 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.730613 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.730582 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.730777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.730586 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.736780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.736756 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsknq\" (UniqueName: \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\") pod \"maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:21.806117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:21.806091 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:22.262886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:22.259173 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 22:42:22.338878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:22.335581 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" event={"ID":"affc590b-ffd9-44a5-bba1-91c84c67cccb","Type":"ContainerStarted","Data":"6fc8f9bb4c229c98a916d65d32819e88370ea491a4ecadd3521b53d0cb08e379"} Nov 01 22:42:23.357429 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:23.357397 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" event={"ID":"affc590b-ffd9-44a5-bba1-91c84c67cccb","Type":"ContainerStarted","Data":"a882a3c21ecaebefaf84d6ede555c2360830b5e255fdb9606cef4ab19dc7ef25"} Nov 01 22:42:23.357869 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:23.357521 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:23.381425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:23.381376 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" podStartSLOduration=1.949249085 podStartE2EDuration="2.381361766s" totalImagesPullingTime="432.112681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:23.376563577 +0000 UTC m=+1113.152550405" watchObservedRunningTime="2026-11-01 22:42:23.381361766 +0000 UTC m=+1113.157348617" Nov 01 22:42:26.322362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:26.322331 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 22:42:29.373557 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:29.373531 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 22:42:29.921872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:29.921827 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 22:42:29.927642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:29.927616 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:29.966989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:29.966951 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 22:42:29.992572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:29.992542 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-075771-openshift-default-dockercfg-ht2bd\"" Nov 01 22:42:30.067068 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067025 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067155 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067184 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067358 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85vf\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067426 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067461 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067553 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067514 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067610 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067551 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.067677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.067662 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.177033 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.176920 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.177821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.177796 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.178296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.178279 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.178992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.178968 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85vf\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.179487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.179467 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.236877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.235930 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.236877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.236012 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.236877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.236057 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.236877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.236192 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.237221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.236998 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.237221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.178235 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.237318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.183294 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.246186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.237578 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.246186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.177692 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.246186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.186300 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.256912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.256363 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85vf\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.279525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.267746 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.279525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.271722 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.293982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.289222 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:30.862729 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:42:30.862684 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6109756_fe25_4c5e_8ed2_0d42ad4bc9e1.slice/crio-bc58af184fec17a9049874692b2d3ee3a987d3da1afd07e05c35f116afbcbaa5 WatchSource:0}: Error finding container bc58af184fec17a9049874692b2d3ee3a987d3da1afd07e05c35f116afbcbaa5: Status 404 returned error can't find the container with id bc58af184fec17a9049874692b2d3ee3a987d3da1afd07e05c35f116afbcbaa5 Nov 01 22:42:30.863160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.863056 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 22:42:30.865710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.865677 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:30.865828 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.865742 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:30.865828 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:30.865771 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:31.484932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:31.484132 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerStarted","Data":"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a"} Nov 01 22:42:31.484932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:31.484211 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerStarted","Data":"bc58af184fec17a9049874692b2d3ee3a987d3da1afd07e05c35f116afbcbaa5"} Nov 01 22:42:31.511476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:31.511414 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podStartSLOduration=2.511382512 podStartE2EDuration="2.511382512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:31.507246567 +0000 UTC m=+1121.283233409" watchObservedRunningTime="2026-11-01 22:42:31.511382512 +0000 UTC m=+1121.287369342" Nov 01 22:42:32.294887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:32.293333 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:32.314535 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:32.314008 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:32.494999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:32.494950 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:32.495894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:32.495839 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:42:45.933074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.933038 5778 generic.go:356] "Generic (PLEG): container finished" podID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerID="37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6" exitCode=0 Nov 01 22:42:45.933599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.933201 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerDied","Data":"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6"} Nov 01 22:42:45.933599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.933560 5778 scope.go:122] "RemoveContainer" containerID="37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6" Nov 01 22:42:45.935430 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.935393 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:45.935552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.935490 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:45.935552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:45.935537 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:47.002872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:46.994212 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerStarted","Data":"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde"} Nov 01 22:42:47.545878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:47.541221 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:47.546221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:47.546195 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:42:47.546416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:47.546390 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 22:42:48.539877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:48.536361 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:42:48.539877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:48.536441 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 22:42:48.830715 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:42:48.830620 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:42:49.539358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:49.539319 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:42:49.539706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:49.539674 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 22:42:50.536045 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:50.535994 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:42:50.536701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:50.536667 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 22:42:51.545879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:51.535233 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:42:51.545879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:51.535287 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 22:42:52.560958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:52.560915 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:52.561475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:52.561383 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:52.562350 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:52.562330 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 22:42:59.829150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:59.829115 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-790fc9"] Nov 01 22:42:59.835911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:59.835884 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:42:59.840184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:59.840158 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 22:42:59.840983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:59.840957 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-790fc9"] Nov 01 22:42:59.948117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:42:59.948070 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncgn\" (UniqueName: \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\") pod \"test-curl-6983-790fc9\" (UID: \"fddb856a-803c-478b-a423-1c3c9a5dba14\") " pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:00.049778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:00.049741 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncgn\" (UniqueName: \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\") pod \"test-curl-6983-790fc9\" (UID: \"fddb856a-803c-478b-a423-1c3c9a5dba14\") " pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:00.059150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:00.059111 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncgn\" (UniqueName: \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\") pod \"test-curl-6983-790fc9\" (UID: \"fddb856a-803c-478b-a423-1c3c9a5dba14\") " pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:00.152966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:00.152877 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:00.466033 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:43:00.466004 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfddb856a_803c_478b_a423_1c3c9a5dba14.slice/crio-9b2000756c6d6ea9da7e0aa633117aa2a99138aa7d6eb8b6f3d04175e3c497a4 WatchSource:0}: Error finding container 9b2000756c6d6ea9da7e0aa633117aa2a99138aa7d6eb8b6f3d04175e3c497a4: Status 404 returned error can't find the container with id 9b2000756c6d6ea9da7e0aa633117aa2a99138aa7d6eb8b6f3d04175e3c497a4 Nov 01 22:43:00.466212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:00.466187 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-790fc9"] Nov 01 22:43:01.171326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:01.171293 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-790fc9" event={"ID":"fddb856a-803c-478b-a423-1c3c9a5dba14","Type":"ContainerStarted","Data":"741c7a179b667ee5c3b4f4b117cf0e34d3680f511274b15f9bb6b69725fb1ab0"} Nov 01 22:43:01.171326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:01.171333 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-790fc9" event={"ID":"fddb856a-803c-478b-a423-1c3c9a5dba14","Type":"ContainerStarted","Data":"9b2000756c6d6ea9da7e0aa633117aa2a99138aa7d6eb8b6f3d04175e3c497a4"} Nov 01 22:43:01.188151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:01.188100 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-790fc9" podStartSLOduration=2.188087254 podStartE2EDuration="2.188087254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:01.186539625 +0000 UTC m=+1150.962526453" watchObservedRunningTime="2026-11-01 22:43:01.188087254 +0000 UTC m=+1150.964074084" Nov 01 22:43:02.116713 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.116637 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-790fc9"] Nov 01 22:43:02.123125 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.122753 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-790fc9"] Nov 01 22:43:02.298262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.298088 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-4a6a78"] Nov 01 22:43:02.299246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.299218 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" containerName="test-curl-6983-790fc9" Nov 01 22:43:02.299246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.299245 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" containerName="test-curl-6983-790fc9" Nov 01 22:43:02.299486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.299415 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" containerName="test-curl-6983-790fc9" Nov 01 22:43:02.304808 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.304785 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-4a6a78"] Nov 01 22:43:02.304977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.304893 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:02.349724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.349537 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh4t\" (UniqueName: \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\") pod \"test-curl-6983-4a6a78\" (UID: \"2b6347c7-c227-423d-83c3-ed9f1f2b72fa\") " pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:02.451904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.451864 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh4t\" (UniqueName: \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\") pod \"test-curl-6983-4a6a78\" (UID: \"2b6347c7-c227-423d-83c3-ed9f1f2b72fa\") " pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:02.460806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.460771 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh4t\" (UniqueName: \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\") pod \"test-curl-6983-4a6a78\" (UID: \"2b6347c7-c227-423d-83c3-ed9f1f2b72fa\") " pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:02.623640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.623607 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:02.988764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:02.981553 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-4a6a78"] Nov 01 22:43:03.260325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:03.260210 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-4a6a78" event={"ID":"2b6347c7-c227-423d-83c3-ed9f1f2b72fa","Type":"ContainerStarted","Data":"f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36"} Nov 01 22:43:03.260325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:03.260270 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-4a6a78" event={"ID":"2b6347c7-c227-423d-83c3-ed9f1f2b72fa","Type":"ContainerStarted","Data":"fb2caba718f60f7275337e8a87147170cdcd3e5794d83ede9ccf7a5f07fa20e9"} Nov 01 22:43:03.260564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:03.260322 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-790fc9" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" containerName="test-curl-6983-790fc9" containerID="cri-o://741c7a179b667ee5c3b4f4b117cf0e34d3680f511274b15f9bb6b69725fb1ab0" gracePeriod=2 Nov 01 22:43:03.289886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:03.278800 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-4a6a78" podStartSLOduration=1.2787808680000001 podStartE2EDuration="1.278780868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:03.274406051 +0000 UTC m=+1153.050392882" watchObservedRunningTime="2026-11-01 22:43:03.278780868 +0000 UTC m=+1153.054767698" Nov 01 22:43:03.986876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:03.985902 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-4a6a78"] Nov 01 22:43:04.038224 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.031460 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-4a6a78"] Nov 01 22:43:04.109163 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.109132 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-7bca2b"] Nov 01 22:43:04.109796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.109783 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" containerName="test-curl-6983-4a6a78" Nov 01 22:43:04.109841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.109798 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" containerName="test-curl-6983-4a6a78" Nov 01 22:43:04.109959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.109948 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" containerName="test-curl-6983-4a6a78" Nov 01 22:43:04.113795 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.113771 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:04.120247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.120221 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-7bca2b"] Nov 01 22:43:04.192739 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.192705 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdfl\" (UniqueName: \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\") pod \"test-curl-6983-7bca2b\" (UID: \"ae5c7970-87d1-48c9-a1bd-64554bcf9035\") " pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:04.319339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.319237 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdfl\" (UniqueName: \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\") pod \"test-curl-6983-7bca2b\" (UID: \"ae5c7970-87d1-48c9-a1bd-64554bcf9035\") " pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:04.328987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.328955 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdfl\" (UniqueName: \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\") pod \"test-curl-6983-7bca2b\" (UID: \"ae5c7970-87d1-48c9-a1bd-64554bcf9035\") " pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:04.452499 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.452455 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:04.735820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:04.735788 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-7bca2b"] Nov 01 22:43:04.736001 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:43:04.735796 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae5c7970_87d1_48c9_a1bd_64554bcf9035.slice/crio-f1183a7155fac3b30e1fb3a3ad1d8c4613a29b4ae286de7e5254028446e345ef WatchSource:0}: Error finding container f1183a7155fac3b30e1fb3a3ad1d8c4613a29b4ae286de7e5254028446e345ef: Status 404 returned error can't find the container with id f1183a7155fac3b30e1fb3a3ad1d8c4613a29b4ae286de7e5254028446e345ef Nov 01 22:43:05.278426 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.278396 5778 generic.go:356] "Generic (PLEG): container finished" podID="fddb856a-803c-478b-a423-1c3c9a5dba14" containerID="741c7a179b667ee5c3b4f4b117cf0e34d3680f511274b15f9bb6b69725fb1ab0" exitCode=137 Nov 01 22:43:05.280327 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.280295 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-7bca2b" event={"ID":"ae5c7970-87d1-48c9-a1bd-64554bcf9035","Type":"ContainerStarted","Data":"0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484"} Nov 01 22:43:05.280435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.280342 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-7bca2b" event={"ID":"ae5c7970-87d1-48c9-a1bd-64554bcf9035","Type":"ContainerStarted","Data":"f1183a7155fac3b30e1fb3a3ad1d8c4613a29b4ae286de7e5254028446e345ef"} Nov 01 22:43:05.280435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.280386 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-4a6a78" podUID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" containerName="test-curl-6983-4a6a78" containerID="cri-o://f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36" gracePeriod=2 Nov 01 22:43:05.300123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.300063 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-7bca2b" podStartSLOduration=1.3000485259999999 podStartE2EDuration="1.300048526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:05.296369682 +0000 UTC m=+1155.072356512" watchObservedRunningTime="2026-11-01 22:43:05.300048526 +0000 UTC m=+1155.076035353" Nov 01 22:43:05.412107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.412079 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:05.542793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.542712 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\" (UniqueName: \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\") pod \"fddb856a-803c-478b-a423-1c3c9a5dba14\" (UID: \"fddb856a-803c-478b-a423-1c3c9a5dba14\") " Nov 01 22:43:05.545104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.545079 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn" pod "fddb856a-803c-478b-a423-1c3c9a5dba14" (UID: "fddb856a-803c-478b-a423-1c3c9a5dba14"). InnerVolumeSpecName "kube-api-access-kncgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:05.645328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.645289 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kncgn\" (UniqueName: \"kubernetes.io/projected/fddb856a-803c-478b-a423-1c3c9a5dba14-kube-api-access-kncgn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:05.974266 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.974209 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-7bca2b"] Nov 01 22:43:05.981688 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:05.980812 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-7bca2b"] Nov 01 22:43:06.089966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.089936 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-9165cb"] Nov 01 22:43:06.090632 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.090618 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" containerName="test-curl-6983-7bca2b" Nov 01 22:43:06.090684 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.090633 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" containerName="test-curl-6983-7bca2b" Nov 01 22:43:06.090799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.090789 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" containerName="test-curl-6983-7bca2b" Nov 01 22:43:06.094434 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.094417 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:06.107690 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.106461 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-9165cb"] Nov 01 22:43:06.155271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.155239 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284fp\" (UniqueName: \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\") pod \"test-curl-6983-9165cb\" (UID: \"bd60e28f-1001-4412-a0a9-28539b0c8086\") " pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:06.257258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.257167 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284fp\" (UniqueName: \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\") pod \"test-curl-6983-9165cb\" (UID: \"bd60e28f-1001-4412-a0a9-28539b0c8086\") " pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:06.269077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.269048 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284fp\" (UniqueName: \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\") pod \"test-curl-6983-9165cb\" (UID: \"bd60e28f-1001-4412-a0a9-28539b0c8086\") " pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:06.292676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.292642 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:06.292676 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.292683 5778 scope.go:122] "RemoveContainer" containerID="741c7a179b667ee5c3b4f4b117cf0e34d3680f511274b15f9bb6b69725fb1ab0" Nov 01 22:43:06.306045 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:06.306007 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-790fc9\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" pod="opendatahub/test-curl-6983-790fc9" Nov 01 22:43:06.427880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.411075 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:06.458114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.458080 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddb856a-803c-478b-a423-1c3c9a5dba14" path="/var/lib/kubelet/pods/fddb856a-803c-478b-a423-1c3c9a5dba14/volumes" Nov 01 22:43:06.909298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:06.909263 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-9165cb"] Nov 01 22:43:06.911045 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:43:06.911015 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd60e28f_1001_4412_a0a9_28539b0c8086.slice/crio-fd53401d21deec1dd7c441bb8c6a85125838372010e7400674146c7c03c171d0 WatchSource:0}: Error finding container fd53401d21deec1dd7c441bb8c6a85125838372010e7400674146c7c03c171d0: Status 404 returned error can't find the container with id fd53401d21deec1dd7c441bb8c6a85125838372010e7400674146c7c03c171d0 Nov 01 22:43:07.339383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.337966 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-9165cb" event={"ID":"bd60e28f-1001-4412-a0a9-28539b0c8086","Type":"ContainerStarted","Data":"a2c7d8c647d5f79b279b2dbf136b72423990ec1684b917ed8836c99a6b2d6d99"} Nov 01 22:43:07.339383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.338095 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-9165cb" event={"ID":"bd60e28f-1001-4412-a0a9-28539b0c8086","Type":"ContainerStarted","Data":"fd53401d21deec1dd7c441bb8c6a85125838372010e7400674146c7c03c171d0"} Nov 01 22:43:07.353527 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.353486 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-7bca2b" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" containerName="test-curl-6983-7bca2b" containerID="cri-o://0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484" gracePeriod=2 Nov 01 22:43:07.622248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.622220 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:07.708563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.708517 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\" (UniqueName: \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\") pod \"2b6347c7-c227-423d-83c3-ed9f1f2b72fa\" (UID: \"2b6347c7-c227-423d-83c3-ed9f1f2b72fa\") " Nov 01 22:43:07.710968 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.710930 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t" pod "2b6347c7-c227-423d-83c3-ed9f1f2b72fa" (UID: "2b6347c7-c227-423d-83c3-ed9f1f2b72fa"). InnerVolumeSpecName "kube-api-access-srh4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:07.821576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:07.811815 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh4t\" (UniqueName: \"kubernetes.io/projected/2b6347c7-c227-423d-83c3-ed9f1f2b72fa-kube-api-access-srh4t\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:08.429876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.407238 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" path="/var/lib/kubelet/pods/2b6347c7-c227-423d-83c3-ed9f1f2b72fa/volumes" Nov 01 22:43:08.439906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.432193 5778 generic.go:356] "Generic (PLEG): container finished" podID="2b6347c7-c227-423d-83c3-ed9f1f2b72fa" containerID="f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36" exitCode=137 Nov 01 22:43:08.439906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.433313 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-4a6a78" Nov 01 22:43:08.467895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.443152 5778 scope.go:122] "RemoveContainer" containerID="f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36" Nov 01 22:43:08.512725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.496540 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-9165cb" podStartSLOduration=2.496515648 podStartE2EDuration="2.496515648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:07.381350706 +0000 UTC m=+1157.157337563" watchObservedRunningTime="2026-11-01 22:43:08.496515648 +0000 UTC m=+1158.272502477" Nov 01 22:43:08.606121 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.598206 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-9165cb"] Nov 01 22:43:08.606121 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.605052 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-9165cb"] Nov 01 22:43:08.618704 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:08.615229 5778 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6347c7_c227_423d_83c3_ed9f1f2b72fa.slice/crio-fb2caba718f60f7275337e8a87147170cdcd3e5794d83ede9ccf7a5f07fa20e9\": RecentStats: unable to find data in memory cache]" Nov 01 22:43:08.680881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.680837 5778 scope.go:122] "RemoveContainer" containerID="f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36" Nov 01 22:43:08.681436 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:08.681415 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36\": container with ID starting with f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36 not found: ID does not exist" containerID="f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36" Nov 01 22:43:08.681590 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:08.681565 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36"} err="failed to get container status \"f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36\": rpc error: code = NotFound desc = could not find container \"f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36\": container with ID starting with f5a1ec74fb79426b5e099d1ba171e20d9c8b769bd3ae79a9b349a0fea083bb36 not found: ID does not exist" Nov 01 22:43:09.469922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:09.466553 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-9165cb" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" containerName="test-curl-6983-9165cb" containerID="cri-o://a2c7d8c647d5f79b279b2dbf136b72423990ec1684b917ed8836c99a6b2d6d99" gracePeriod=2 Nov 01 22:43:09.482231 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:09.482191 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-9165cb\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:10.130865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:10.129697 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.195167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:10.193954 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.440877 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:10.429984 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-9165cb\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:12.145029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.144830 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:43:12.215177 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.180284 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.215177 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.215140 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.247888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.245483 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.532007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.528078 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:12.532007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.529151 5778 generic.go:356] "Generic (PLEG): container finished" podID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" containerID="0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484" exitCode=137 Nov 01 22:43:12.540498 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:12.533068 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bca2b\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:12.560981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.560955 5778 scope.go:122] "RemoveContainer" containerID="0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484" Nov 01 22:43:12.635812 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.635788 5778 scope.go:122] "RemoveContainer" containerID="0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484" Nov 01 22:43:12.636466 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:12.636438 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484\": container with ID starting with 0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484 not found: ID does not exist" containerID="0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484" Nov 01 22:43:12.636663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.636599 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484"} err="failed to get container status \"0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484\": rpc error: code = NotFound desc = could not find container \"0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484\": container with ID starting with 0fbf4fdb366117059bc096984096101d565b14cf80edf335fda35f29c12f0484 not found: ID does not exist" Nov 01 22:43:12.740933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.724795 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\" (UniqueName: \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\") pod \"ae5c7970-87d1-48c9-a1bd-64554bcf9035\" (UID: \"ae5c7970-87d1-48c9-a1bd-64554bcf9035\") " Nov 01 22:43:12.761993 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.761956 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl" pod "ae5c7970-87d1-48c9-a1bd-64554bcf9035" (UID: "ae5c7970-87d1-48c9-a1bd-64554bcf9035"). InnerVolumeSpecName "kube-api-access-qrdfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:12.847375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:12.847259 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdfl\" (UniqueName: \"kubernetes.io/projected/ae5c7970-87d1-48c9-a1bd-64554bcf9035-kube-api-access-qrdfl\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:13.025938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.025888 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 01 22:43:13.222689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.200319 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:43:13.222689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.215548 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:43:13.259137 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.247933 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:13.259137 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.248342 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:43:13.294314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.278719 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:43:13.294314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.285571 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:43:13.294314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.286536 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:43:13.294314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.286875 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:43:13.294314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.287177 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:43:13.465970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.465941 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:43:13.995878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:13.983830 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:43:14.044878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.044059 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:43:14.068877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.050362 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:14.095531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.095497 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:43:14.263012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.248487 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:14.263012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.249979 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:43:14.424589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.424544 5778 generic.go:356] "Generic (PLEG): container finished" podID="bd60e28f-1001-4412-a0a9-28539b0c8086" containerID="a2c7d8c647d5f79b279b2dbf136b72423990ec1684b917ed8836c99a6b2d6d99" exitCode=137 Nov 01 22:43:14.492877 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:14.460671 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bca2b\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:14.492877 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:14.489016 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bca2b\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:14.511168 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:14.511124 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bca2b\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" pod="opendatahub/test-curl-6983-7bca2b" Nov 01 22:43:14.597173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.572379 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5c7970-87d1-48c9-a1bd-64554bcf9035" path="/var/lib/kubelet/pods/ae5c7970-87d1-48c9-a1bd-64554bcf9035/volumes" Nov 01 22:43:14.691025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.690991 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:43:14.797289 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.795730 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:14.906585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.906549 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\" (UniqueName: \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\") pod \"bd60e28f-1001-4412-a0a9-28539b0c8086\" (UID: \"bd60e28f-1001-4412-a0a9-28539b0c8086\") " Nov 01 22:43:14.908983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:14.908952 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp" pod "bd60e28f-1001-4412-a0a9-28539b0c8086" (UID: "bd60e28f-1001-4412-a0a9-28539b0c8086"). InnerVolumeSpecName "kube-api-access-284fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:15.009357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:15.009323 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284fp\" (UniqueName: \"kubernetes.io/projected/bd60e28f-1001-4412-a0a9-28539b0c8086-kube-api-access-284fp\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:15.476785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:15.476758 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:15.477358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:15.476926 5778 scope.go:122] "RemoveContainer" containerID="a2c7d8c647d5f79b279b2dbf136b72423990ec1684b917ed8836c99a6b2d6d99" Nov 01 22:43:15.493371 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:15.493339 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-9165cb\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" pod="opendatahub/test-curl-6983-9165cb" Nov 01 22:43:16.378378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:16.378297 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" path="/var/lib/kubelet/pods/bd60e28f-1001-4412-a0a9-28539b0c8086/volumes" Nov 01 22:43:17.534269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.534230 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:43:17.535115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.535096 5778 cpu_manager.go:414] "containerMap: removing container" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" containerName="test-curl-6983-9165cb" Nov 01 22:43:17.535115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.535115 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" containerName="test-curl-6983-9165cb" Nov 01 22:43:17.535262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.535226 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd60e28f-1001-4412-a0a9-28539b0c8086" containerName="test-curl-6983-9165cb" Nov 01 22:43:17.546916 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.546887 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.560093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.560062 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-30578128-serving-cert\"" Nov 01 22:43:17.563335 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.563305 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:43:17.615565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.615521 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.615723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.615581 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhxk\" (UniqueName: \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.615723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.615633 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.615885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.615819 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.722321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.720028 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.722321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.720086 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhxk\" (UniqueName: \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.722321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.720138 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.722321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.720349 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.722321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.722296 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.753601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.753469 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.753601 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.753540 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.774731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.774695 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhxk\" (UniqueName: \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\") pod \"maas-api-e2e-mt-30578128-65d96db648-sbb7w\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:17.914926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:17.914891 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:18.368122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:18.351795 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:43:18.372617 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:43:18.372580 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a7721d_061b_4919_ba8b_43ad6b119f46.slice/crio-a137ebd937e902f004f58c4e260097a7cecdc5837fff7fa424d24826cf2767c3 WatchSource:0}: Error finding container a137ebd937e902f004f58c4e260097a7cecdc5837fff7fa424d24826cf2767c3: Status 404 returned error can't find the container with id a137ebd937e902f004f58c4e260097a7cecdc5837fff7fa424d24826cf2767c3 Nov 01 22:43:18.515736 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:18.515661 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" event={"ID":"81a7721d-061b-4919-ba8b-43ad6b119f46","Type":"ContainerStarted","Data":"a137ebd937e902f004f58c4e260097a7cecdc5837fff7fa424d24826cf2767c3"} Nov 01 22:43:18.542043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:18.541999 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:18.542543 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:18.542073 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:18.790334 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:18.790293 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:43:19.529180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:19.529145 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" event={"ID":"81a7721d-061b-4919-ba8b-43ad6b119f46","Type":"ContainerStarted","Data":"905982488ebc13348a1d4cc20db82ff4d6f236d1947550b45e9aa5857b29ac9e"} Nov 01 22:43:19.529440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:19.529228 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:19.555641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:19.555589 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" podStartSLOduration=2.081369109 podStartE2EDuration="2.555573549s" totalImagesPullingTime="474.20444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:19.550749021 +0000 UTC m=+1169.326735851" watchObservedRunningTime="2026-11-01 22:43:19.555573549 +0000 UTC m=+1169.331560384" Nov 01 22:43:22.564218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.564182 5778 generic.go:356] "Generic (PLEG): container finished" podID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerID="e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3" exitCode=0 Nov 01 22:43:22.564734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.564257 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerDied","Data":"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3"} Nov 01 22:43:22.564842 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.564825 5778 scope.go:122] "RemoveContainer" containerID="e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3" Nov 01 22:43:22.566214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566190 5778 generic.go:356] "Generic (PLEG): container finished" podID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerID="87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a" exitCode=0 Nov 01 22:43:22.566297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566273 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerDied","Data":"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a"} Nov 01 22:43:22.566721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566704 5778 scope.go:122] "RemoveContainer" containerID="87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a" Nov 01 22:43:22.566816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566790 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:22.566913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566881 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:22.566965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.566917 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:22.568416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.568390 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:22.568498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.568454 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:22.568498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:22.568483 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:23.580258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:23.580211 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerStarted","Data":"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08"} Nov 01 22:43:23.583027 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:23.582994 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerStarted","Data":"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9"} Nov 01 22:43:24.270342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.270307 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:43:24.270744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.270718 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.270991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.270965 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:24.290248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.290216 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:43:24.290482 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.290364 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.290626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:24.290500 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:25.288880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:25.286950 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.288880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:25.287024 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:25.290329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:25.290301 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.290495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:25.290469 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:25.577189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:25.577062 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:43:26.245877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:26.245316 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:26.245877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:26.245382 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:26.290830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:26.290794 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:26.291259 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:26.290864 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:27.244876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.244321 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:27.244876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.244386 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:27.290882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.290487 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:27.290882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.290544 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:27.407957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.407922 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:43:27.479879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.459070 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:43:27.479879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.459204 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.512173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.512092 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4534ad-openshift-default-dockercfg-f4x9q\"" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.651624 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.651686 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w8q\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.651720 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.651827 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.651872 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.652179 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.652214 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.652278 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.842176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.652333 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.871876 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.871923 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872146 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872176 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872252 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872301 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872376 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872412 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w8q\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872443 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.872982 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.883392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.873312 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.950564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.943662 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.950564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.943957 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:27.950564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:27.944178 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.010390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.010353 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.064973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.064946 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.068612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.068551 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.097923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.090198 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:43:28.133879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.129758 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w8q\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\") pod \"e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.171377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.171258 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:28.253895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.245302 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:28.253895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.245377 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:28.317677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.317632 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:28.318978 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.318344 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:28.896256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:28.894627 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:43:29.221878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.203035 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:29.221878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.203143 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:29.221878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.203188 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:29.262577 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.261780 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:29.262577 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.261841 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:29.354770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.353017 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:29.354770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.353098 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:29.396794 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:29.369113 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerStarted","Data":"c3357c3540d5c8543c2eed371f2dd710091a465f6032cf8261587093663be41f"} Nov 01 22:43:30.241980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:30.241518 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 22:43:30.241980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:30.241576 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 22:43:30.290378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:30.290298 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:43:30.290691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:30.290635 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 22:43:30.526937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:30.524669 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerStarted","Data":"ba358bd93104c4849d220bbafa479942a0c993a7a032d6b2a5a9a9af85fd72a5"} Nov 01 22:43:31.172569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.172536 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:31.178804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.178775 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:31.211624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.211566 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podStartSLOduration=4.21154994 podStartE2EDuration="4.21154994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:30.571347713 +0000 UTC m=+1180.347334589" watchObservedRunningTime="2026-11-01 22:43:31.21154994 +0000 UTC m=+1180.987536768" Nov 01 22:43:31.293586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.292244 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:43:31.294573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.294537 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 22:43:31.297578 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.297550 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:43:31.297869 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.297827 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:43:31.298630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.298612 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 22:43:31.561418 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.561335 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:31.562373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:31.562350 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:43:32.461897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.461841 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:43:32.472583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.472552 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.478730 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.478691 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:43:32.485708 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.485685 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30578128\"/\"openshift-service-ca.crt\"" Nov 01 22:43:32.486305 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.486276 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30578128\"/\"e2e-lifecycle-model-30578128-kserve-self-signed-certs\"" Nov 01 22:43:32.486533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.486518 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30578128\"/\"kube-root-ca.crt\"" Nov 01 22:43:32.486754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.486725 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30578128\"/\"default-dockercfg-rn7q2\"" Nov 01 22:43:32.500612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.500581 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvzn\" (UniqueName: \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.500943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.500915 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.501131 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.501111 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.501431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.501395 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.501752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.501716 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.502054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.502034 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.603331 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603287 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.603837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603427 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.603837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603555 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvzn\" (UniqueName: \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.603837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603713 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.603837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603775 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.604170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.603936 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.604240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.604167 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.604367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.604334 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.604556 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.604504 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.608267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.608238 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.608455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.608438 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.612571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.612536 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvzn\" (UniqueName: \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\") pod \"e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:32.825987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:32.814225 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:33.308202 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:43:33.308173 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54daeafc_beff_43f2_b420_e24468782220.slice/crio-ae37b6c78687b72df7c41b1bc4db310bb2c3f7ca0c625b8a4cc1af540b1f6bb6 WatchSource:0}: Error finding container ae37b6c78687b72df7c41b1bc4db310bb2c3f7ca0c625b8a4cc1af540b1f6bb6: Status 404 returned error can't find the container with id ae37b6c78687b72df7c41b1bc4db310bb2c3f7ca0c625b8a4cc1af540b1f6bb6 Nov 01 22:43:33.310124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:33.310098 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:43:33.731939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:33.730510 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" event={"ID":"54daeafc-beff-43f2-b420-e24468782220","Type":"ContainerStarted","Data":"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934"} Nov 01 22:43:33.731939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:33.730560 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" event={"ID":"54daeafc-beff-43f2-b420-e24468782220","Type":"ContainerStarted","Data":"ae37b6c78687b72df7c41b1bc4db310bb2c3f7ca0c625b8a4cc1af540b1f6bb6"} Nov 01 22:43:33.810884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:33.810460 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" podStartSLOduration=1.810440201 podStartE2EDuration="1.810440201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:33.798425722 +0000 UTC m=+1183.574412552" watchObservedRunningTime="2026-11-01 22:43:33.810440201 +0000 UTC m=+1183.586427030" Nov 01 22:43:33.823959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:33.815718 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:42.814986 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:42.814944 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:42.830569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:42.830540 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:42.970360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:42.970326 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:43:48.775989 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:43:48.775950 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:43:50.574165 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:50.574106 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:43:50.576724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:50.576698 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:43:52.506540 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:43:52.506501 5778 scope.go:122] "RemoveContainer" containerID="a52c7df1750b0dd02b0f7efc2195b667b4998bf86e4436fac662268193c0da1c" Nov 01 22:44:05.983014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:05.982975 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 22:44:05.992752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:05.992712 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.007807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.007772 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a9f94e-openshift-default-dockercfg-t57nk\"" Nov 01 22:44:06.013423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.013391 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 22:44:06.085750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085602 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.085750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085663 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrhw\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.085750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085691 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085812 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085908 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085950 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.085998 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.086031 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.086442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.086117 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.187943 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188039 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188090 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188138 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188201 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188237 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188346 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188518 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.188894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.188578 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrhw\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.189514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.189470 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.195791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.190204 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.195791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.190720 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.195791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.190975 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.195791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.191198 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.195791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.195087 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.208875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.198776 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrhw\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.213014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.212615 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.217875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.217811 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.364924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.357163 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:06.980907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:06.979115 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 22:44:07.001242 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:44:07.001211 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d157576_5447_4574_8903_4cc537809016.slice/crio-ea1b888809b11b4e17b7682338662aeb01ceb9299a983d17dd911ab363044b5a WatchSource:0}: Error finding container ea1b888809b11b4e17b7682338662aeb01ceb9299a983d17dd911ab363044b5a: Status 404 returned error can't find the container with id ea1b888809b11b4e17b7682338662aeb01ceb9299a983d17dd911ab363044b5a Nov 01 22:44:07.016096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.016017 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:07.016324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.016117 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:07.017076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.016165 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:07.427368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.427326 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerStarted","Data":"b85aceedd9d9cd5428d5e60f80115a7a2bb50e2460c0590c8d8ff620c08bb078"} Nov 01 22:44:07.427557 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.427409 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerStarted","Data":"ea1b888809b11b4e17b7682338662aeb01ceb9299a983d17dd911ab363044b5a"} Nov 01 22:44:07.457823 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:07.457756 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podStartSLOduration=2.457733846 podStartE2EDuration="2.457733846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:07.451737172 +0000 UTC m=+1217.227724025" watchObservedRunningTime="2026-11-01 22:44:07.457733846 +0000 UTC m=+1217.233720676" Nov 01 22:44:08.438878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:08.438273 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:08.598876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:08.593392 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:09.482648 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:09.482608 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:09.485397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:09.485372 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 22:44:18.931877 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:44:18.923078 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:44:19.715652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:19.714959 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.984936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:19.973678 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" podUID="54daeafc-beff-43f2-b420-e24468782220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:44:20.089051 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.063984 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:20.513267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.513216 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:44:20.772257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.772169 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:44:20.779709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.779679 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:44:20.781229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.780027 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:20.787536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.787512 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-df4b43-serving-cert\"" Nov 01 22:44:20.933929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.933374 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zq4\" (UniqueName: \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:20.933929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.933452 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:20.933929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.933516 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:20.933929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:20.933871 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.041976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.039139 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.041976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.039383 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.041976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.039556 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zq4\" (UniqueName: \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.041976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.039607 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.041976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.039652 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.076129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.068908 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.076129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.073821 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zq4\" (UniqueName: \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.097887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.079068 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\") pod \"maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:21.263697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:21.263642 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:22.097453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:22.093246 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:44:22.708078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:22.708038 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" event={"ID":"da921265-e497-4051-b3f8-854d16fe008e","Type":"ContainerStarted","Data":"a9d230ef91e7af25fb306ddf28f6d1bbf7a3990aabf89adc3ed687e4bbbbe8ce"} Nov 01 22:44:23.718023 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:23.717982 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" event={"ID":"da921265-e497-4051-b3f8-854d16fe008e","Type":"ContainerStarted","Data":"76a6d1e571a495b4eda1c6bc43ff3f2843bc7da61011403d249f9bb4dec2e196"} Nov 01 22:44:23.718536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:23.718135 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:23.746679 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:23.746615 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" podStartSLOduration=4.157226213 podStartE2EDuration="4.74659588s" totalImagesPullingTime="589.369667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:23.739021656 +0000 UTC m=+1233.515008487" watchObservedRunningTime="2026-11-01 22:44:23.74659588 +0000 UTC m=+1233.522582710" Nov 01 22:44:27.760108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.760074 5778 generic.go:356] "Generic (PLEG): container finished" podID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerID="ba358bd93104c4849d220bbafa479942a0c993a7a032d6b2a5a9a9af85fd72a5" exitCode=0 Nov 01 22:44:27.760579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.760147 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerDied","Data":"ba358bd93104c4849d220bbafa479942a0c993a7a032d6b2a5a9a9af85fd72a5"} Nov 01 22:44:27.760798 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.760775 5778 scope.go:122] "RemoveContainer" containerID="ba358bd93104c4849d220bbafa479942a0c993a7a032d6b2a5a9a9af85fd72a5" Nov 01 22:44:27.762834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.762800 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:27.762990 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.762911 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:27.762990 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:27.762962 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:44:28.781671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:28.781621 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerStarted","Data":"bbb4aeefb156cedb88e99cbe6e967f35e61abd2e9bdae102aa5d9043532dfc6a"} Nov 01 22:44:29.173229 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.173196 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:44:29.173449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.173327 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:29.173449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.173376 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:29.768187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.768155 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:44:29.770530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.770505 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:44:29.776098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.776074 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.776427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.776402 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:44:29.781184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.780025 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a5c3234-serving-cert\"" Nov 01 22:44:29.861291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.861261 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.861741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.861433 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wmb\" (UniqueName: \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.861741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.861532 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.861741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.861667 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.975132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.972264 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.975132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.972442 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wmb\" (UniqueName: \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.975132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.972484 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.975132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.972606 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:29.975132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:29.973698 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:30.025554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.000973 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:30.036876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.027533 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:30.052097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.041541 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wmb\" (UniqueName: \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\") pod \"maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:30.109880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.109561 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:30.174810 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.174734 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:30.174810 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.174804 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:30.780279 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:44:30.780201 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf106a314_90c8_4b03_be66_c3b7b37399ef.slice/crio-8a09bb5a02db252fd33456a72bd2e1e18a76fa020f39ff9ff11b7fb5c05c54a1 WatchSource:0}: Error finding container 8a09bb5a02db252fd33456a72bd2e1e18a76fa020f39ff9ff11b7fb5c05c54a1: Status 404 returned error can't find the container with id 8a09bb5a02db252fd33456a72bd2e1e18a76fa020f39ff9ff11b7fb5c05c54a1 Nov 01 22:44:30.797773 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.797738 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:44:30.814966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:30.814926 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" event={"ID":"f106a314-90c8-4b03-be66-c3b7b37399ef","Type":"ContainerStarted","Data":"8a09bb5a02db252fd33456a72bd2e1e18a76fa020f39ff9ff11b7fb5c05c54a1"} Nov 01 22:44:31.173956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.173921 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:31.174408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.173977 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:31.562238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.562136 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:44:31.934877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.907611 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" event={"ID":"f106a314-90c8-4b03-be66-c3b7b37399ef","Type":"ContainerStarted","Data":"dfc4ac8fe130cbba898d3f6fdab458ce8339857bc8c8f38f90daae4de7ce506d"} Nov 01 22:44:31.934877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.922357 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:31.966423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:31.966358 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" podStartSLOduration=2.462684266 podStartE2EDuration="2.96633638s" totalImagesPullingTime="503.652114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:31.9515308 +0000 UTC m=+1241.727517630" watchObservedRunningTime="2026-11-01 22:44:31.96633638 +0000 UTC m=+1241.742323212" Nov 01 22:44:32.172569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:32.172531 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:32.172946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:32.172837 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:33.172471 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:33.172432 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:33.172950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:33.172491 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:34.172797 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:34.172754 5778 patch_prober.go:29] interesting pod/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:44:34.173293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:34.172819 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 01 22:44:35.177674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:35.177637 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:44:35.178932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:35.178914 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:44:38.997035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:38.997006 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:44:48.776593 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:44:48.776554 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:44:55.838600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:55.837975 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:55.838600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:55.838142 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:55.922879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:55.918970 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:55.922879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:44:55.919036 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:00.363604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.363485 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:00.371523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.371495 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:00.371665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.371618 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.375813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.375787 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.403375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.403347 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:00.414455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.414424 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:00.415448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.415408 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:45:00.470565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.470526 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.470782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.470706 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvj5g\" (UniqueName: \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.470878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.470830 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fdq\" (UniqueName: \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.470987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.470962 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.572156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.572127 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.572415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.572195 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvj5g\" (UniqueName: \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.572415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.572247 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fdq\" (UniqueName: \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.572415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.572273 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.572898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.572829 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.573079 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.573057 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.594463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.582313 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fdq\" (UniqueName: \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.594463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.582690 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvj5g\" (UniqueName: \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\") pod \"maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:00.702107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.702070 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:00.713734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:00.713705 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:01.009809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:01.009780 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:01.011316 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:45:01.011286 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6dc20f_9149_4ecd_a5c7_75a5fef7a2f2.slice/crio-66befd41c2c464e125f9b6cd4452884a453c0e5cf109a87ebc64dabda2535036 WatchSource:0}: Error finding container 66befd41c2c464e125f9b6cd4452884a453c0e5cf109a87ebc64dabda2535036: Status 404 returned error can't find the container with id 66befd41c2c464e125f9b6cd4452884a453c0e5cf109a87ebc64dabda2535036 Nov 01 22:45:01.035569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:01.035538 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:01.373708 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:01.373601 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" event={"ID":"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2","Type":"ContainerStarted","Data":"66befd41c2c464e125f9b6cd4452884a453c0e5cf109a87ebc64dabda2535036"} Nov 01 22:45:01.375253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:01.375219 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerStarted","Data":"bd43ad98531a249bc2ff7ab8015ac9fb48b2143757faf8cece6871018326d720"} Nov 01 22:45:02.384492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.384401 5778 generic.go:356] "Generic (PLEG): container finished" podID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerID="1fe122ade8d417ae4585d7f22b944476451cf5b7618378ab0c857cf416182ba4" exitCode=60 Nov 01 22:45:02.385022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.384484 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" event={"ID":"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2","Type":"ContainerDied","Data":"1fe122ade8d417ae4585d7f22b944476451cf5b7618378ab0c857cf416182ba4"} Nov 01 22:45:02.385022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.384901 5778 scope.go:122] "RemoveContainer" containerID="1fe122ade8d417ae4585d7f22b944476451cf5b7618378ab0c857cf416182ba4" Nov 01 22:45:02.386450 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.386426 5778 generic.go:356] "Generic (PLEG): container finished" podID="e3819829-b1ed-4ef5-985e-44345c783160" containerID="0f7bf572a016d8ec8dace013c8da5662b11a34637e953c0f7f6744205ab67d4e" exitCode=60 Nov 01 22:45:02.386572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.386500 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerDied","Data":"0f7bf572a016d8ec8dace013c8da5662b11a34637e953c0f7f6744205ab67d4e"} Nov 01 22:45:02.386761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:02.386743 5778 scope.go:122] "RemoveContainer" containerID="0f7bf572a016d8ec8dace013c8da5662b11a34637e953c0f7f6744205ab67d4e" Nov 01 22:45:03.060939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.060770 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-e085c8"] Nov 01 22:45:03.079796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.078377 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-e085c8"] Nov 01 22:45:03.079796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.078473 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:03.085366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.085337 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 22:45:03.134318 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.134274 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm7z\" (UniqueName: \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\") pod \"test-curl-6968-e085c8\" (UID: \"febaf165-ca4d-4e7c-8dec-b1f62397060d\") " pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:03.237064 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.237024 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm7z\" (UniqueName: \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\") pod \"test-curl-6968-e085c8\" (UID: \"febaf165-ca4d-4e7c-8dec-b1f62397060d\") " pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:03.248108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.248072 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm7z\" (UniqueName: \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\") pod \"test-curl-6968-e085c8\" (UID: \"febaf165-ca4d-4e7c-8dec-b1f62397060d\") " pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:03.399195 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.399155 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:03.408463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.407478 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerStarted","Data":"1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5"} Nov 01 22:45:03.411298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.411264 5778 generic.go:356] "Generic (PLEG): container finished" podID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerID="e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921" exitCode=60 Nov 01 22:45:03.411475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.411367 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" event={"ID":"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2","Type":"ContainerDied","Data":"e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921"} Nov 01 22:45:03.411475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.411426 5778 scope.go:122] "RemoveContainer" containerID="1fe122ade8d417ae4585d7f22b944476451cf5b7618378ab0c857cf416182ba4" Nov 01 22:45:03.411791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.411768 5778 scope.go:122] "RemoveContainer" containerID="e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921" Nov 01 22:45:03.413213 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:03.412274 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg_odh-ai-gateway-infra(0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" Nov 01 22:45:03.449767 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.449712 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" podStartSLOduration=3.449692486 podStartE2EDuration="3.449692486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.447656533 +0000 UTC m=+1273.223643362" watchObservedRunningTime="2026-11-01 22:45:03.449692486 +0000 UTC m=+1273.225679314" Nov 01 22:45:03.731210 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:03.731181 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-e085c8"] Nov 01 22:45:03.732064 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:45:03.732030 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfebaf165_ca4d_4e7c_8dec_b1f62397060d.slice/crio-6be913e68ebb187f0e218b62756b6a60949588d91e8a7a076eb2fd5625b5a699 WatchSource:0}: Error finding container 6be913e68ebb187f0e218b62756b6a60949588d91e8a7a076eb2fd5625b5a699: Status 404 returned error can't find the container with id 6be913e68ebb187f0e218b62756b6a60949588d91e8a7a076eb2fd5625b5a699 Nov 01 22:45:04.422754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.422725 5778 scope.go:122] "RemoveContainer" containerID="e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921" Nov 01 22:45:04.423818 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:04.423786 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg_odh-ai-gateway-infra(0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" Nov 01 22:45:04.424529 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.424500 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-e085c8" event={"ID":"febaf165-ca4d-4e7c-8dec-b1f62397060d","Type":"ContainerStarted","Data":"df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3"} Nov 01 22:45:04.424637 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.424547 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-e085c8" event={"ID":"febaf165-ca4d-4e7c-8dec-b1f62397060d","Type":"ContainerStarted","Data":"6be913e68ebb187f0e218b62756b6a60949588d91e8a7a076eb2fd5625b5a699"} Nov 01 22:45:04.426815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.426792 5778 generic.go:356] "Generic (PLEG): container finished" podID="e3819829-b1ed-4ef5-985e-44345c783160" containerID="1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5" exitCode=60 Nov 01 22:45:04.426945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.426911 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerDied","Data":"1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5"} Nov 01 22:45:04.427011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.426964 5778 scope.go:122] "RemoveContainer" containerID="0f7bf572a016d8ec8dace013c8da5662b11a34637e953c0f7f6744205ab67d4e" Nov 01 22:45:04.427230 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.427209 5778 scope.go:122] "RemoveContainer" containerID="1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5" Nov 01 22:45:04.427650 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:04.427624 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf_odh-ai-gateway-infra(e3819829-b1ed-4ef5-985e-44345c783160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" podUID="e3819829-b1ed-4ef5-985e-44345c783160" Nov 01 22:45:04.482831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:04.482768 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-e085c8" podStartSLOduration=1.482751953 podStartE2EDuration="1.482751953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.478065607 +0000 UTC m=+1274.254052438" watchObservedRunningTime="2026-11-01 22:45:04.482751953 +0000 UTC m=+1274.258738824" Nov 01 22:45:05.134498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:05.134462 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-e085c8"] Nov 01 22:45:05.150922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:05.148843 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-e085c8"] Nov 01 22:45:05.440227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:05.440200 5778 scope.go:122] "RemoveContainer" containerID="1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5" Nov 01 22:45:05.440665 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:05.440642 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf_odh-ai-gateway-infra(e3819829-b1ed-4ef5-985e-44345c783160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" podUID="e3819829-b1ed-4ef5-985e-44345c783160" Nov 01 22:45:06.482442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:06.482384 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-e085c8" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" containerName="test-curl-6968-e085c8" containerID="cri-o://df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3" gracePeriod=2 Nov 01 22:45:08.752170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:08.752099 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:08.824339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:08.824300 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\" (UniqueName: \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\") pod \"febaf165-ca4d-4e7c-8dec-b1f62397060d\" (UID: \"febaf165-ca4d-4e7c-8dec-b1f62397060d\") " Nov 01 22:45:08.846380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:08.846345 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z" pod "febaf165-ca4d-4e7c-8dec-b1f62397060d" (UID: "febaf165-ca4d-4e7c-8dec-b1f62397060d"). InnerVolumeSpecName "kube-api-access-fdm7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:08.927253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:08.927210 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm7z\" (UniqueName: \"kubernetes.io/projected/febaf165-ca4d-4e7c-8dec-b1f62397060d-kube-api-access-fdm7z\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:09.511202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:09.511158 5778 generic.go:356] "Generic (PLEG): container finished" podID="febaf165-ca4d-4e7c-8dec-b1f62397060d" containerID="df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3" exitCode=137 Nov 01 22:45:09.511410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:09.511229 5778 scope.go:122] "RemoveContainer" containerID="df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3" Nov 01 22:45:09.511410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:09.511254 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:09.525415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:09.525389 5778 scope.go:122] "RemoveContainer" containerID="df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3" Nov 01 22:45:09.525754 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:09.525725 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3\": container with ID starting with df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3 not found: ID does not exist" containerID="df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3" Nov 01 22:45:09.525845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:09.525766 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3"} err="failed to get container status \"df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3\": rpc error: code = NotFound desc = could not find container \"df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3\": container with ID starting with df3e372ea3446ce320a3c584a017878ace5ace1c961d5ba6f2608e72f8f26bc3 not found: ID does not exist" Nov 01 22:45:09.526790 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:09.526769 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-e085c8\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:10.405876 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:10.401340 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-e085c8\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" pod="opendatahub/test-curl-6968-e085c8" Nov 01 22:45:10.419646 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:10.419607 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" path="/var/lib/kubelet/pods/febaf165-ca4d-4e7c-8dec-b1f62397060d/volumes" Nov 01 22:45:14.367814 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:14.367781 5778 scope.go:122] "RemoveContainer" containerID="e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921" Nov 01 22:45:15.367250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:15.367206 5778 scope.go:122] "RemoveContainer" containerID="1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5" Nov 01 22:45:15.571000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:15.570965 5778 generic.go:356] "Generic (PLEG): container finished" podID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerID="a929a9a36826f057dae7a30573429722e2ddf9bef2c053e5abc51c37f894f674" exitCode=60 Nov 01 22:45:15.571476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:15.571042 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" event={"ID":"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2","Type":"ContainerDied","Data":"a929a9a36826f057dae7a30573429722e2ddf9bef2c053e5abc51c37f894f674"} Nov 01 22:45:15.571476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:15.571104 5778 scope.go:122] "RemoveContainer" containerID="e4f2c81aee2d9a6c9bcd1c5fdb370b993131fd600d858e6e814352225d8c4921" Nov 01 22:45:15.571619 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:15.571599 5778 scope.go:122] "RemoveContainer" containerID="a929a9a36826f057dae7a30573429722e2ddf9bef2c053e5abc51c37f894f674" Nov 01 22:45:15.572079 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:15.572043 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg_odh-ai-gateway-infra(0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" Nov 01 22:45:16.600804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.600768 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerStarted","Data":"1e6a60e5a5ce2868faf1e3a8d7c9597cf2721bb815f037b11c7b85e20aca6cf4"} Nov 01 22:45:16.608908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.608874 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:16.803709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.803677 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:16.839238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.839205 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\" (UniqueName: \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\") pod \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " Nov 01 22:45:16.839238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.839246 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\") pod \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\" (UID: \"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2\") " Nov 01 22:45:16.839678 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.839657 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca" pod "0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" (UID: "0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:16.840190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.840114 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:16.841916 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.841883 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq" pod "0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" (UID: "0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2"). InnerVolumeSpecName "kube-api-access-m7fdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:16.942491 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:16.942448 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7fdq\" (UniqueName: \"kubernetes.io/projected/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2-kube-api-access-m7fdq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:17.610500 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.610465 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" Nov 01 22:45:17.610500 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.610491 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg" event={"ID":"0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2","Type":"ContainerDied","Data":"66befd41c2c464e125f9b6cd4452884a453c0e5cf109a87ebc64dabda2535036"} Nov 01 22:45:17.611084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.610556 5778 scope.go:122] "RemoveContainer" containerID="a929a9a36826f057dae7a30573429722e2ddf9bef2c053e5abc51c37f894f674" Nov 01 22:45:17.612803 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.612775 5778 generic.go:356] "Generic (PLEG): container finished" podID="e3819829-b1ed-4ef5-985e-44345c783160" containerID="1e6a60e5a5ce2868faf1e3a8d7c9597cf2721bb815f037b11c7b85e20aca6cf4" exitCode=60 Nov 01 22:45:17.612970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.612872 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerDied","Data":"1e6a60e5a5ce2868faf1e3a8d7c9597cf2721bb815f037b11c7b85e20aca6cf4"} Nov 01 22:45:17.613422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.613402 5778 scope.go:122] "RemoveContainer" containerID="1e6a60e5a5ce2868faf1e3a8d7c9597cf2721bb815f037b11c7b85e20aca6cf4" Nov 01 22:45:17.613836 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:17.613814 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf_odh-ai-gateway-infra(e3819829-b1ed-4ef5-985e-44345c783160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" podUID="e3819829-b1ed-4ef5-985e-44345c783160" Nov 01 22:45:17.624378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.624354 5778 scope.go:122] "RemoveContainer" containerID="1cac0fdf12ffe4acf01382ba560d6b7a6d859935bc7b8f561fa3e1cbb17c90a5" Nov 01 22:45:17.641239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.641206 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:17.667141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.667102 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:17.675966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:17.675936 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892885-7tbbg"] Nov 01 22:45:18.379459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.379415 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" path="/var/lib/kubelet/pods/0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2/volumes" Nov 01 22:45:18.786073 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:18.786031 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:45:18.823597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.823570 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:18.884077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.884031 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\" (UniqueName: \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\") pod \"e3819829-b1ed-4ef5-985e-44345c783160\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " Nov 01 22:45:18.884325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.884261 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\") pod \"e3819829-b1ed-4ef5-985e-44345c783160\" (UID: \"e3819829-b1ed-4ef5-985e-44345c783160\") " Nov 01 22:45:18.885115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.885083 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca" pod "e3819829-b1ed-4ef5-985e-44345c783160" (UID: "e3819829-b1ed-4ef5-985e-44345c783160"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:18.885254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.885226 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3819829-b1ed-4ef5-985e-44345c783160-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:18.887079 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.887050 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g" pod "e3819829-b1ed-4ef5-985e-44345c783160" (UID: "e3819829-b1ed-4ef5-985e-44345c783160"). InnerVolumeSpecName "kube-api-access-vvj5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:18.987886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:18.987780 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvj5g\" (UniqueName: \"kubernetes.io/projected/e3819829-b1ed-4ef5-985e-44345c783160-kube-api-access-vvj5g\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:19.659721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:19.659072 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" event={"ID":"e3819829-b1ed-4ef5-985e-44345c783160","Type":"ContainerDied","Data":"bd43ad98531a249bc2ff7ab8015ac9fb48b2143757faf8cece6871018326d720"} Nov 01 22:45:19.659721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:19.659151 5778 scope.go:122] "RemoveContainer" containerID="1e6a60e5a5ce2868faf1e3a8d7c9597cf2721bb815f037b11c7b85e20aca6cf4" Nov 01 22:45:19.659721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:19.659268 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf" Nov 01 22:45:19.778054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:19.778004 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:19.785591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:19.785558 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2a5c3234-29892885-mwwzf"] Nov 01 22:45:20.379265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:20.379231 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3819829-b1ed-4ef5-985e-44345c783160" path="/var/lib/kubelet/pods/e3819829-b1ed-4ef5-985e-44345c783160/volumes" Nov 01 22:45:27.402008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:27.400978 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:45:27.402008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:27.401249 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" containerID="cri-o://bbb4aeefb156cedb88e99cbe6e967f35e61abd2e9bdae102aa5d9043532dfc6a" gracePeriod=30 Nov 01 22:45:33.060382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.052695 5778 generic.go:356] "Generic (PLEG): container finished" podID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerID="bbb4aeefb156cedb88e99cbe6e967f35e61abd2e9bdae102aa5d9043532dfc6a" exitCode=0 Nov 01 22:45:33.060382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.052792 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerDied","Data":"bbb4aeefb156cedb88e99cbe6e967f35e61abd2e9bdae102aa5d9043532dfc6a"} Nov 01 22:45:33.064671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.063056 5778 scope.go:122] "RemoveContainer" containerID="ba358bd93104c4849d220bbafa479942a0c993a7a032d6b2a5a9a9af85fd72a5" Nov 01 22:45:33.302799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.302765 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:45:33.434356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434317 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.434579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434438 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.434697 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434649 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:33.434878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434799 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.434961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434722 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:33.435021 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.434972 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435023 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435153 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435129 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435172 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435375 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435416 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\") pod \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\" (UID: \"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317\") " Nov 01 22:45:33.435887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.435824 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:33.439834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.437396 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.439834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.437425 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.439834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.437440 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.439834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.437832 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:33.439834 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.439392 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:33.440616 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.440593 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:33.440917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.440649 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:33.440917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.440736 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "kube-api-access-r5w8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:33.442172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.442149 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy" pod "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" (UID: "fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:33.539614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539577 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.539614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539607 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.539614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539617 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w8q\" (UniqueName: \"kubernetes.io/projected/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-kube-api-access-r5w8q\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.539964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539630 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.539964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539642 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.539964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:33.539650 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:34.064532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.064494 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" event={"ID":"fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317","Type":"ContainerDied","Data":"c3357c3540d5c8543c2eed371f2dd710091a465f6032cf8261587093663be41f"} Nov 01 22:45:34.065098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.064562 5778 scope.go:122] "RemoveContainer" containerID="bbb4aeefb156cedb88e99cbe6e967f35e61abd2e9bdae102aa5d9043532dfc6a" Nov 01 22:45:34.065098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.064695 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx" Nov 01 22:45:34.100122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.100083 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:45:34.106986 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.106957 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4534ad-openshift-default-64b6d794f5-qmklx"] Nov 01 22:45:34.379198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:34.379112 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" path="/var/lib/kubelet/pods/fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317/volumes" Nov 01 22:45:47.349901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.349829 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:45:47.350428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.350121 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" podUID="81a7721d-061b-4919-ba8b-43ad6b119f46" containerName="maas-api" containerID="cri-o://905982488ebc13348a1d4cc20db82ff4d6f236d1947550b45e9aa5857b29ac9e" gracePeriod=30 Nov 01 22:45:47.477236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.477194 5778 generic.go:356] "Generic (PLEG): container finished" podID="81a7721d-061b-4919-ba8b-43ad6b119f46" containerID="905982488ebc13348a1d4cc20db82ff4d6f236d1947550b45e9aa5857b29ac9e" exitCode=0 Nov 01 22:45:47.477386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.477264 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" event={"ID":"81a7721d-061b-4919-ba8b-43ad6b119f46","Type":"ContainerDied","Data":"905982488ebc13348a1d4cc20db82ff4d6f236d1947550b45e9aa5857b29ac9e"} Nov 01 22:45:47.773940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.773208 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:45:47.914062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.911046 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\") pod \"81a7721d-061b-4919-ba8b-43ad6b119f46\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " Nov 01 22:45:47.914062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.911212 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\") pod \"81a7721d-061b-4919-ba8b-43ad6b119f46\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " Nov 01 22:45:47.914062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.911245 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\") pod \"81a7721d-061b-4919-ba8b-43ad6b119f46\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " Nov 01 22:45:47.914062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.911484 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\" (UniqueName: \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\") pod \"81a7721d-061b-4919-ba8b-43ad6b119f46\" (UID: \"81a7721d-061b-4919-ba8b-43ad6b119f46\") " Nov 01 22:45:47.943901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.940397 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca" pod "81a7721d-061b-4919-ba8b-43ad6b119f46" (UID: "81a7721d-061b-4919-ba8b-43ad6b119f46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:48.004878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:47.987090 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls" pod "81a7721d-061b-4919-ba8b-43ad6b119f46" (UID: "81a7721d-061b-4919-ba8b-43ad6b119f46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:48.087660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.086677 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:48.087660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.086711 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a7721d-061b-4919-ba8b-43ad6b119f46-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:48.126726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.122274 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs" pod "81a7721d-061b-4919-ba8b-43ad6b119f46" (UID: "81a7721d-061b-4919-ba8b-43ad6b119f46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:48.126726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.125943 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk" pod "81a7721d-061b-4919-ba8b-43ad6b119f46" (UID: "81a7721d-061b-4919-ba8b-43ad6b119f46"). InnerVolumeSpecName "kube-api-access-vbhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:48.188000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.187962 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a7721d-061b-4919-ba8b-43ad6b119f46-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:48.188000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.188001 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhxk\" (UniqueName: \"kubernetes.io/projected/81a7721d-061b-4919-ba8b-43ad6b119f46-kube-api-access-vbhxk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:48.485862 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.485821 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" Nov 01 22:45:48.485862 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.485840 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w" event={"ID":"81a7721d-061b-4919-ba8b-43ad6b119f46","Type":"ContainerDied","Data":"a137ebd937e902f004f58c4e260097a7cecdc5837fff7fa424d24826cf2767c3"} Nov 01 22:45:48.486394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.485920 5778 scope.go:122] "RemoveContainer" containerID="905982488ebc13348a1d4cc20db82ff4d6f236d1947550b45e9aa5857b29ac9e" Nov 01 22:45:48.543456 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.543419 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:45:48.551462 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:48.551428 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-sbb7w"] Nov 01 22:45:49.080876 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:49.074537 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:45:50.397945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:50.397896 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a7721d-061b-4919-ba8b-43ad6b119f46" path="/var/lib/kubelet/pods/81a7721d-061b-4919-ba8b-43ad6b119f46/volumes" Nov 01 22:45:54.202348 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.202311 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:45:54.219091 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.219027 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" containerID="cri-o://c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979" gracePeriod=30 Nov 01 22:45:54.551524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.551438 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:45:54.552368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552347 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.552368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552368 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552379 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552384 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552418 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552422 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552433 5778 cpu_manager.go:414] "containerMap: removing container" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" containerName="test-curl-6968-e085c8" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552439 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" containerName="test-curl-6968-e085c8" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552458 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552462 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552489 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552494 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552505 5778 cpu_manager.go:414] "containerMap: removing container" podUID="81a7721d-061b-4919-ba8b-43ad6b119f46" containerName="maas-api" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552510 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a7721d-061b-4919-ba8b-43ad6b119f46" containerName="maas-api" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552530 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552534 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552640 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552648 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552654 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="febaf165-ca4d-4e7c-8dec-b1f62397060d" containerName="test-curl-6968-e085c8" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552663 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552669 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6dc20f-9149-4ecd-a5c7-75a5fef7a2f2" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552676 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a7721d-061b-4919-ba8b-43ad6b119f46" containerName="maas-api" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552683 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.552829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.552688 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:45:54.558549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.558529 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.586332 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.585635 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:45:54.618778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.618748 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2476f8-serving-cert\"" Nov 01 22:45:54.780884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.762252 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.781310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.781263 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrdt\" (UniqueName: \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.781709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.781687 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.781933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.781915 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.885567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.885469 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.885567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.885521 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrdt\" (UniqueName: \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.885818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.885596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.885818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.885622 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.885818 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:54.885644 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-serving-cert: secret "maas-api-e2e-api-a-2476f8-serving-cert" not found Nov 01 22:45:54.885818 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:45:54.885733 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls podName:2f007021-12ac-4dab-bce9-758ad3556e6e nodeName:}" failed. No retries permitted until 2026-11-01 22:45:55.385709022 +0000 UTC m=+1325.161695841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls") pod "maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" (UID: "2f007021-12ac-4dab-bce9-758ad3556e6e") : secret "maas-api-e2e-api-a-2476f8-serving-cert" not found Nov 01 22:45:54.886546 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.886520 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.940214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.940179 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:54.940560 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:54.940536 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrdt\" (UniqueName: \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:55.417884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:55.411911 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:55.461879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:55.460977 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:55.484877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:55.484124 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:56.782880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:56.773218 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:45:56.809465 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:56.802640 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:45:56.853894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:56.848984 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" event={"ID":"2f007021-12ac-4dab-bce9-758ad3556e6e","Type":"ContainerStarted","Data":"c2f86d96ea02dbbf661ee484ba3846c030becf2231328ac094316e69f9aaedb1"} Nov 01 22:45:59.055813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.055125 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" event={"ID":"2f007021-12ac-4dab-bce9-758ad3556e6e","Type":"ContainerStarted","Data":"730f32a7163efdcd7d77ff5baaf4a278ce89170252cf2ccdc054e821ae8e24f7"} Nov 01 22:45:59.055813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.055332 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:45:59.111940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.111844 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" podStartSLOduration=4.069026082 podStartE2EDuration="5.111824488s" totalImagesPullingTime="1.042798406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:59.100990637 +0000 UTC m=+1328.876977490" watchObservedRunningTime="2026-11-01 22:45:59.111824488 +0000 UTC m=+1328.887811320" Nov 01 22:45:59.727275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.727208 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:45:59.744746 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.744714 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.744949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.744929 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.744966 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745007 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.745119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745090 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745148 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745231 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745215 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.745286 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745270 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745334 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745385 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745416 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\") pod \"004d2649-2fc7-49ca-a1ab-489fb6f12218\" (UID: \"004d2649-2fc7-49ca-a1ab-489fb6f12218\") " Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745595 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745775 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.745727 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.745959 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.746533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.746251 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.746533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.746274 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.746533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.746287 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.746533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.746299 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.746533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.746312 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d2649-2fc7-49ca-a1ab-489fb6f12218-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.747752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.747722 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "kube-api-access-bbk9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.748261 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.748235 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:59.748439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.748422 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.748914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.748844 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token" pod "004d2649-2fc7-49ca-a1ab-489fb6f12218" (UID: "004d2649-2fc7-49ca-a1ab-489fb6f12218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.847988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.847956 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.847988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.847985 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.847988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.847995 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d2649-2fc7-49ca-a1ab-489fb6f12218-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.848239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:45:59.848004 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbk9n\" (UniqueName: \"kubernetes.io/projected/004d2649-2fc7-49ca-a1ab-489fb6f12218-kube-api-access-bbk9n\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.120985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.117475 5778 generic.go:356] "Generic (PLEG): container finished" podID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerID="c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979" exitCode=0 Nov 01 22:46:00.120985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.118721 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" Nov 01 22:46:00.125351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.124794 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" event={"ID":"004d2649-2fc7-49ca-a1ab-489fb6f12218","Type":"ContainerDied","Data":"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979"} Nov 01 22:46:00.125351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.124942 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk" event={"ID":"004d2649-2fc7-49ca-a1ab-489fb6f12218","Type":"ContainerDied","Data":"24b0b8869bc5380a94b102fcb517e17d6c4b246248f64128b88f4b36e6f1f097"} Nov 01 22:46:00.125351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.124976 5778 scope.go:122] "RemoveContainer" containerID="c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979" Nov 01 22:46:00.176011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.171543 5778 scope.go:122] "RemoveContainer" containerID="c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979" Nov 01 22:46:00.176011 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:00.173333 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979\": container with ID starting with c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979 not found: ID does not exist" containerID="c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979" Nov 01 22:46:00.176011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.173365 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979"} err="failed to get container status \"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979\": rpc error: code = NotFound desc = could not find container \"c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979\": container with ID starting with c58a170ca6cde7c8abf7fde87feafb7c2156a1f0c2f738e6e7079012c5195979 not found: ID does not exist" Nov 01 22:46:00.190351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.190315 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:46:00.196502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.196463 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qfnlk"] Nov 01 22:46:00.377414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:00.377324 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" path="/var/lib/kubelet/pods/004d2649-2fc7-49ca-a1ab-489fb6f12218/volumes" Nov 01 22:46:04.701878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:04.701535 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:46:04.701878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:04.701781 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" podUID="da921265-e497-4051-b3f8-854d16fe008e" containerName="maas-api" containerID="cri-o://76a6d1e571a495b4eda1c6bc43ff3f2843bc7da61011403d249f9bb4dec2e196" gracePeriod=30 Nov 01 22:46:04.748455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:04.748412 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" podUID="da921265-e497-4051-b3f8-854d16fe008e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.107:8443/health\": dial tcp 10.132.0.107:8443: connect: connection refused" Nov 01 22:46:05.309604 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.309211 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:46:05.425872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.423469 5778 generic.go:356] "Generic (PLEG): container finished" podID="da921265-e497-4051-b3f8-854d16fe008e" containerID="76a6d1e571a495b4eda1c6bc43ff3f2843bc7da61011403d249f9bb4dec2e196" exitCode=0 Nov 01 22:46:05.425872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.423750 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" event={"ID":"da921265-e497-4051-b3f8-854d16fe008e","Type":"ContainerDied","Data":"76a6d1e571a495b4eda1c6bc43ff3f2843bc7da61011403d249f9bb4dec2e196"} Nov 01 22:46:05.503709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.503677 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:46:05.587843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.587761 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\" (UniqueName: \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\") pod \"da921265-e497-4051-b3f8-854d16fe008e\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " Nov 01 22:46:05.588031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.587967 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\") pod \"da921265-e497-4051-b3f8-854d16fe008e\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " Nov 01 22:46:05.588031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.588024 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\") pod \"da921265-e497-4051-b3f8-854d16fe008e\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " Nov 01 22:46:05.588171 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.588155 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\") pod \"da921265-e497-4051-b3f8-854d16fe008e\" (UID: \"da921265-e497-4051-b3f8-854d16fe008e\") " Nov 01 22:46:05.588406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.588378 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca" pod "da921265-e497-4051-b3f8-854d16fe008e" (UID: "da921265-e497-4051-b3f8-854d16fe008e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:05.588898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.588838 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da921265-e497-4051-b3f8-854d16fe008e-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.590323 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.590301 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls" pod "da921265-e497-4051-b3f8-854d16fe008e" (UID: "da921265-e497-4051-b3f8-854d16fe008e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:05.590323 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.590313 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4" pod "da921265-e497-4051-b3f8-854d16fe008e" (UID: "da921265-e497-4051-b3f8-854d16fe008e"). InnerVolumeSpecName "kube-api-access-n5zq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:05.590443 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.590317 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs" pod "da921265-e497-4051-b3f8-854d16fe008e" (UID: "da921265-e497-4051-b3f8-854d16fe008e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:05.690866 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.690816 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zq4\" (UniqueName: \"kubernetes.io/projected/da921265-e497-4051-b3f8-854d16fe008e-kube-api-access-n5zq4\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.691024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.690892 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.691024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:05.690904 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da921265-e497-4051-b3f8-854d16fe008e-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:06.439023 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.438979 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" event={"ID":"da921265-e497-4051-b3f8-854d16fe008e","Type":"ContainerDied","Data":"a9d230ef91e7af25fb306ddf28f6d1bbf7a3990aabf89adc3ed687e4bbbbe8ce"} Nov 01 22:46:06.439494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.439033 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v" Nov 01 22:46:06.439494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.439082 5778 scope.go:122] "RemoveContainer" containerID="76a6d1e571a495b4eda1c6bc43ff3f2843bc7da61011403d249f9bb4dec2e196" Nov 01 22:46:06.462719 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.462687 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:46:06.471787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.471762 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df4b43-7b4d56bfcf-cmj6v"] Nov 01 22:46:06.954362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.954308 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 22:46:06.955519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955488 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:46:06.955519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955519 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955541 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955550 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955575 5778 cpu_manager.go:414] "containerMap: removing container" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955583 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955628 5778 cpu_manager.go:414] "containerMap: removing container" podUID="da921265-e497-4051-b3f8-854d16fe008e" containerName="maas-api" Nov 01 22:46:06.955741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955635 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da921265-e497-4051-b3f8-854d16fe008e" containerName="maas-api" Nov 01 22:46:06.956066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955818 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3819829-b1ed-4ef5-985e-44345c783160" containerName="cleanup" Nov 01 22:46:06.956066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955836 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="004d2649-2fc7-49ca-a1ab-489fb6f12218" containerName="istio-proxy" Nov 01 22:46:06.956066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955869 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4f8cb6-cc0d-4100-a05e-1a8d05d9f317" containerName="istio-proxy" Nov 01 22:46:06.956066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.955882 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="da921265-e497-4051-b3f8-854d16fe008e" containerName="maas-api" Nov 01 22:46:06.966575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.966543 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:06.974928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.974900 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 22:46:06.985528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:06.985501 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-c6cec6-serving-cert\"" Nov 01 22:46:07.112243 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.112208 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.112427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.112285 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.112427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.112393 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.112516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.112505 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptq5\" (UniqueName: \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.219930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.216816 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.219930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.216960 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.219930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.217024 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.219930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.217092 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptq5\" (UniqueName: \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.219930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.219613 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.228191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.224320 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.234348 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.234315 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptq5\" (UniqueName: \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.236475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.236444 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.290898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.289001 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:07.667326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:07.667284 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 22:46:08.379615 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.379582 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da921265-e497-4051-b3f8-854d16fe008e" path="/var/lib/kubelet/pods/da921265-e497-4051-b3f8-854d16fe008e/volumes" Nov 01 22:46:08.466563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.466461 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" event={"ID":"1ddf4019-13be-4213-937a-78ef85850d51","Type":"ContainerStarted","Data":"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1"} Nov 01 22:46:08.466563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.466525 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" event={"ID":"1ddf4019-13be-4213-937a-78ef85850d51","Type":"ContainerStarted","Data":"14a4f37c508a72078627209bd514f036b833df59b065fad79c603cde1fbdea97"} Nov 01 22:46:08.466776 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.466708 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:08.486089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.486019 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" podStartSLOduration=2.003076922 podStartE2EDuration="2.486004526s" totalImagesPullingTime="482.927604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:08.485555908 +0000 UTC m=+1338.261542738" watchObservedRunningTime="2026-11-01 22:46:08.486004526 +0000 UTC m=+1338.261991371" Nov 01 22:46:08.987421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.987388 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:08.992515 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:08.992491 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.015067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.015035 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-30578128-serving-cert\"" Nov 01 22:46:09.015067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.015058 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:09.027974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.027905 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9hn\" (UniqueName: \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.028251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.027982 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.028251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.028121 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.028390 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.028283 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.136686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.136648 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.136976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.136785 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.138134 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:09.138112 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-30578128-serving-cert: secret "maas-api-e2e-mt-30578128-serving-cert" not found Nov 01 22:46:09.138339 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:09.138324 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls podName:c538925b-8ca8-4d0e-a327-388de4153935 nodeName:}" failed. No retries permitted until 2026-11-01 22:46:09.63830099 +0000 UTC m=+1339.414287797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls") pod "maas-api-e2e-mt-30578128-65d96db648-bsdzm" (UID: "c538925b-8ca8-4d0e-a327-388de4153935") : secret "maas-api-e2e-mt-30578128-serving-cert" not found Nov 01 22:46:09.145959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.145920 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.156325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.146384 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9hn\" (UniqueName: \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.158377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.158190 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.172525 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.172491 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9hn\" (UniqueName: \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.179310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.179278 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.669074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.669033 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.671747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.671721 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") pod \"maas-api-e2e-mt-30578128-65d96db648-bsdzm\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:09.919926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:09.917256 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:10.379643 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:46:10.379610 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc538925b_8ca8_4d0e_a327_388de4153935.slice/crio-b7fac9f093655e44bf49da16c76d2898593ab9a9a7242965cb25a3f6c10109a5 WatchSource:0}: Error finding container b7fac9f093655e44bf49da16c76d2898593ab9a9a7242965cb25a3f6c10109a5: Status 404 returned error can't find the container with id b7fac9f093655e44bf49da16c76d2898593ab9a9a7242965cb25a3f6c10109a5 Nov 01 22:46:10.389716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:10.389685 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:10.483498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:10.483454 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" event={"ID":"c538925b-8ca8-4d0e-a327-388de4153935","Type":"ContainerStarted","Data":"b7fac9f093655e44bf49da16c76d2898593ab9a9a7242965cb25a3f6c10109a5"} Nov 01 22:46:11.500586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:11.500463 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" event={"ID":"c538925b-8ca8-4d0e-a327-388de4153935","Type":"ContainerStarted","Data":"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6"} Nov 01 22:46:11.500586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:11.500544 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:11.525467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:11.525417 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" podStartSLOduration=3.095596207 podStartE2EDuration="3.525402541s" totalImagesPullingTime="429.806334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:11.52147777 +0000 UTC m=+1341.297464602" watchObservedRunningTime="2026-11-01 22:46:11.525402541 +0000 UTC m=+1341.301389369" Nov 01 22:46:14.483970 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:14.483934 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 22:46:17.513938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:17.513904 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:18.776145 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:18.776105 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:46:33.195105 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.195022 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-a616eb"] Nov 01 22:46:33.200149 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.200125 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:33.209299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.209221 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 22:46:33.213734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.213705 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a616eb"] Nov 01 22:46:33.279093 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.279055 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sggm\" (UniqueName: \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\") pod \"test-curl-6968-a616eb\" (UID: \"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013\") " pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:33.380891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.380844 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sggm\" (UniqueName: \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\") pod \"test-curl-6968-a616eb\" (UID: \"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013\") " pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:33.389606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.389578 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sggm\" (UniqueName: \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\") pod \"test-curl-6968-a616eb\" (UID: \"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013\") " pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:33.524929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:33.522840 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:34.153634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.153602 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a616eb"] Nov 01 22:46:34.626273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.626158 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:46:34.627107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.627054 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" podUID="f106a314-90c8-4b03-be66-c3b7b37399ef" containerName="maas-api" containerID="cri-o://dfc4ac8fe130cbba898d3f6fdab458ce8339857bc8c8f38f90daae4de7ce506d" gracePeriod=30 Nov 01 22:46:34.837040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.836767 5778 generic.go:356] "Generic (PLEG): container finished" podID="f106a314-90c8-4b03-be66-c3b7b37399ef" containerID="dfc4ac8fe130cbba898d3f6fdab458ce8339857bc8c8f38f90daae4de7ce506d" exitCode=0 Nov 01 22:46:34.837040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.836945 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" event={"ID":"f106a314-90c8-4b03-be66-c3b7b37399ef","Type":"ContainerDied","Data":"dfc4ac8fe130cbba898d3f6fdab458ce8339857bc8c8f38f90daae4de7ce506d"} Nov 01 22:46:34.840785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.839558 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a616eb" event={"ID":"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013","Type":"ContainerStarted","Data":"1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af"} Nov 01 22:46:34.840785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:34.839602 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a616eb" event={"ID":"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013","Type":"ContainerStarted","Data":"0799696e581971c6ceddc796321c0f471d89764b790d604669c186e71b27f633"} Nov 01 22:46:35.728432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.728382 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:46:35.786449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.786336 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-a616eb" podStartSLOduration=2.786316023 podStartE2EDuration="2.786316023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:34.899054211 +0000 UTC m=+1364.675041052" watchObservedRunningTime="2026-11-01 22:46:35.786316023 +0000 UTC m=+1365.562302852" Nov 01 22:46:35.940357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.929455 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\") pod \"f106a314-90c8-4b03-be66-c3b7b37399ef\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " Nov 01 22:46:35.940357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.929507 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\") pod \"f106a314-90c8-4b03-be66-c3b7b37399ef\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " Nov 01 22:46:35.940357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.929549 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\") pod \"f106a314-90c8-4b03-be66-c3b7b37399ef\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " Nov 01 22:46:35.940357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.932300 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\" (UniqueName: \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\") pod \"f106a314-90c8-4b03-be66-c3b7b37399ef\" (UID: \"f106a314-90c8-4b03-be66-c3b7b37399ef\") " Nov 01 22:46:35.951974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.950746 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca" pod "f106a314-90c8-4b03-be66-c3b7b37399ef" (UID: "f106a314-90c8-4b03-be66-c3b7b37399ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:35.959923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.959176 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" Nov 01 22:46:35.959923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.959548 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7" event={"ID":"f106a314-90c8-4b03-be66-c3b7b37399ef","Type":"ContainerDied","Data":"8a09bb5a02db252fd33456a72bd2e1e18a76fa020f39ff9ff11b7fb5c05c54a1"} Nov 01 22:46:35.959923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.959602 5778 scope.go:122] "RemoveContainer" containerID="dfc4ac8fe130cbba898d3f6fdab458ce8339857bc8c8f38f90daae4de7ce506d" Nov 01 22:46:35.976881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.975755 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb" pod "f106a314-90c8-4b03-be66-c3b7b37399ef" (UID: "f106a314-90c8-4b03-be66-c3b7b37399ef"). InnerVolumeSpecName "kube-api-access-82wmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:35.976881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.975837 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs" pod "f106a314-90c8-4b03-be66-c3b7b37399ef" (UID: "f106a314-90c8-4b03-be66-c3b7b37399ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:35.993399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:35.992643 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls" pod "f106a314-90c8-4b03-be66-c3b7b37399ef" (UID: "f106a314-90c8-4b03-be66-c3b7b37399ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:36.036933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.035109 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106a314-90c8-4b03-be66-c3b7b37399ef-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:36.036933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.035144 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:36.036933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.035159 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f106a314-90c8-4b03-be66-c3b7b37399ef-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:36.036933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.035171 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wmb\" (UniqueName: \"kubernetes.io/projected/f106a314-90c8-4b03-be66-c3b7b37399ef-kube-api-access-82wmb\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:36.248110 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.248071 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-a616eb"] Nov 01 22:46:36.276326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.272201 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-a616eb"] Nov 01 22:46:36.303251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.303158 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:46:36.314057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.314028 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a5c3234-677d455bcc-jqxc7"] Nov 01 22:46:36.379808 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.379768 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f106a314-90c8-4b03-be66-c3b7b37399ef" path="/var/lib/kubelet/pods/f106a314-90c8-4b03-be66-c3b7b37399ef/volumes" Nov 01 22:46:36.970256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:36.970210 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-a616eb" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" containerName="test-curl-6968-a616eb" containerID="cri-o://1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af" gracePeriod=2 Nov 01 22:46:39.291752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:39.291727 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:39.304641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:39.304608 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\" (UniqueName: \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\") pod \"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013\" (UID: \"b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013\") " Nov 01 22:46:39.307198 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:39.307159 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm" pod "b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" (UID: "b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013"). InnerVolumeSpecName "kube-api-access-7sggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:39.407835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:39.407789 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sggm\" (UniqueName: \"kubernetes.io/projected/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013-kube-api-access-7sggm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.081083 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.081035 5778 generic.go:356] "Generic (PLEG): container finished" podID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" containerID="1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af" exitCode=137 Nov 01 22:46:40.081302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.081240 5778 scope.go:122] "RemoveContainer" containerID="1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af" Nov 01 22:46:40.081411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.081389 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:40.152923 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:40.148694 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a616eb\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:40.164043 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.163635 5778 scope.go:122] "RemoveContainer" containerID="1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af" Nov 01 22:46:40.164364 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:40.164269 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af\": container with ID starting with 1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af not found: ID does not exist" containerID="1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af" Nov 01 22:46:40.164364 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.164307 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af"} err="failed to get container status \"1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af\": rpc error: code = NotFound desc = could not find container \"1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af\": container with ID starting with 1275cd7c1caf269f549d14659aa1be19fe91f447920d720530ebb11fc43794af not found: ID does not exist" Nov 01 22:46:40.406137 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:40.406095 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" path="/var/lib/kubelet/pods/b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013/volumes" Nov 01 22:46:40.409987 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:40.409953 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a616eb\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" pod="opendatahub/test-curl-6968-a616eb" Nov 01 22:46:48.776703 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:48.776654 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:46:49.246155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:49.246121 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:49.246740 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:49.246620 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" podUID="c538925b-8ca8-4d0e-a327-388de4153935" containerName="maas-api" containerID="cri-o://6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6" gracePeriod=30 Nov 01 22:46:50.029451 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.029419 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:50.107895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.107832 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\" (UniqueName: \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\") pod \"c538925b-8ca8-4d0e-a327-388de4153935\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " Nov 01 22:46:50.108087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.108073 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\") pod \"c538925b-8ca8-4d0e-a327-388de4153935\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " Nov 01 22:46:50.108180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.108167 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\") pod \"c538925b-8ca8-4d0e-a327-388de4153935\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " Nov 01 22:46:50.108254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.108239 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") pod \"c538925b-8ca8-4d0e-a327-388de4153935\" (UID: \"c538925b-8ca8-4d0e-a327-388de4153935\") " Nov 01 22:46:50.108580 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.108541 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca" pod "c538925b-8ca8-4d0e-a327-388de4153935" (UID: "c538925b-8ca8-4d0e-a327-388de4153935"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:50.109165 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.109127 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c538925b-8ca8-4d0e-a327-388de4153935-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:50.110278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.110253 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn" pod "c538925b-8ca8-4d0e-a327-388de4153935" (UID: "c538925b-8ca8-4d0e-a327-388de4153935"). InnerVolumeSpecName "kube-api-access-mg9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:50.110377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.110357 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs" pod "c538925b-8ca8-4d0e-a327-388de4153935" (UID: "c538925b-8ca8-4d0e-a327-388de4153935"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:50.110453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.110433 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls" pod "c538925b-8ca8-4d0e-a327-388de4153935" (UID: "c538925b-8ca8-4d0e-a327-388de4153935"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:50.211297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.211217 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:50.211297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.211249 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c538925b-8ca8-4d0e-a327-388de4153935-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:50.211297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.211264 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9hn\" (UniqueName: \"kubernetes.io/projected/c538925b-8ca8-4d0e-a327-388de4153935-kube-api-access-mg9hn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:50.365353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.365321 5778 generic.go:356] "Generic (PLEG): container finished" podID="c538925b-8ca8-4d0e-a327-388de4153935" containerID="6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6" exitCode=0 Nov 01 22:46:50.365536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.365431 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" Nov 01 22:46:50.365536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.365465 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" event={"ID":"c538925b-8ca8-4d0e-a327-388de4153935","Type":"ContainerDied","Data":"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6"} Nov 01 22:46:50.365662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.365535 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm" event={"ID":"c538925b-8ca8-4d0e-a327-388de4153935","Type":"ContainerDied","Data":"b7fac9f093655e44bf49da16c76d2898593ab9a9a7242965cb25a3f6c10109a5"} Nov 01 22:46:50.365662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.365566 5778 scope.go:122] "RemoveContainer" containerID="6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6" Nov 01 22:46:50.379170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.379144 5778 scope.go:122] "RemoveContainer" containerID="6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6" Nov 01 22:46:50.379505 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:46:50.379477 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6\": container with ID starting with 6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6 not found: ID does not exist" containerID="6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6" Nov 01 22:46:50.379570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.379520 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6"} err="failed to get container status \"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6\": rpc error: code = NotFound desc = could not find container \"6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6\": container with ID starting with 6094aa62d2726b044b877ad4b1c13f62b2748c115c0f9b9cf6f282141fae1ae6 not found: ID does not exist" Nov 01 22:46:50.422304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.422274 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:50.446076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:50.446043 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30578128-65d96db648-bsdzm"] Nov 01 22:46:52.378404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:52.378370 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c538925b-8ca8-4d0e-a327-388de4153935" path="/var/lib/kubelet/pods/c538925b-8ca8-4d0e-a327-388de4153935/volumes" Nov 01 22:46:55.292511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:46:55.292022 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:47:01.852032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.851992 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:01.853190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853162 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" containerName="test-curl-6968-a616eb" Nov 01 22:47:01.853329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853190 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" containerName="test-curl-6968-a616eb" Nov 01 22:47:01.853329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853248 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c538925b-8ca8-4d0e-a327-388de4153935" containerName="maas-api" Nov 01 22:47:01.853329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853257 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c538925b-8ca8-4d0e-a327-388de4153935" containerName="maas-api" Nov 01 22:47:01.853329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853310 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f106a314-90c8-4b03-be66-c3b7b37399ef" containerName="maas-api" Nov 01 22:47:01.853329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853320 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f106a314-90c8-4b03-be66-c3b7b37399ef" containerName="maas-api" Nov 01 22:47:01.853570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853484 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f106a314-90c8-4b03-be66-c3b7b37399ef" containerName="maas-api" Nov 01 22:47:01.853570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853504 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d16cc1-d4a0-423a-9ebe-4d1a6dc54013" containerName="test-curl-6968-a616eb" Nov 01 22:47:01.853570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.853519 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c538925b-8ca8-4d0e-a327-388de4153935" containerName="maas-api" Nov 01 22:47:01.861172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.861147 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871207 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871304 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871340 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871433 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fs4\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871482 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871619 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871571 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871627 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871760 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.871914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.871879 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.880579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.877695 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:01.918873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.918813 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-30578128-openshift-default-dockercfg-l7hql\"" Nov 01 22:47:01.974050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974012 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974051 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974120 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fs4\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974149 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974209 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974245 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974329 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974394 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974437 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.974795 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.974777 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.975048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.975024 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.975268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.975241 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.975820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.975761 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.976279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.976256 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.977961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.977938 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.978337 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.978315 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.993355 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.993324 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:01.994494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:01.994468 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fs4\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:02.186709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:02.186671 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:02.755758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:02.755725 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:02.757825 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:47:02.757794 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8d2275_d192_4196_8a07_711f8db86ef3.slice/crio-1593a542d4e3c501f111dedb1693fe0d52e9cc08384127aac732549c87045b57 WatchSource:0}: Error finding container 1593a542d4e3c501f111dedb1693fe0d52e9cc08384127aac732549c87045b57: Status 404 returned error can't find the container with id 1593a542d4e3c501f111dedb1693fe0d52e9cc08384127aac732549c87045b57 Nov 01 22:47:02.761975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:02.761929 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:02.762082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:02.762031 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:02.762128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:02.762077 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:03.558974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:03.558931 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" event={"ID":"8d8d2275-d192-4196-8a07-711f8db86ef3","Type":"ContainerStarted","Data":"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f"} Nov 01 22:47:03.559458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:03.559004 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" event={"ID":"8d8d2275-d192-4196-8a07-711f8db86ef3","Type":"ContainerStarted","Data":"1593a542d4e3c501f111dedb1693fe0d52e9cc08384127aac732549c87045b57"} Nov 01 22:47:03.589664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:03.589605 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podStartSLOduration=2.589592069 podStartE2EDuration="2.589592069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:03.585623017 +0000 UTC m=+1393.361609843" watchObservedRunningTime="2026-11-01 22:47:03.589592069 +0000 UTC m=+1393.365578897" Nov 01 22:47:04.186943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:04.186900 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:04.188312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:04.188288 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:04.188426 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:04.188342 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:05.187609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:05.187571 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:05.188134 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:05.187626 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:06.187032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:06.187000 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:06.187208 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:06.187057 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:07.195027 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:07.194937 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:07.195027 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:07.194997 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:08.193877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:08.188129 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:08.193877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:08.188215 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:09.216547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:09.216007 5778 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:47:09.216547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:09.216101 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 22:47:10.335070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.335023 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" podUID="54daeafc-beff-43f2-b420-e24468782220" containerName="main" containerID="cri-o://30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934" gracePeriod=45 Nov 01 22:47:10.350497 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.349757 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:10.350701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.350645 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:10.352176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.351804 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:10.695061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.695033 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:47:10.705185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.705157 5778 generic.go:356] "Generic (PLEG): container finished" podID="54daeafc-beff-43f2-b420-e24468782220" containerID="30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934" exitCode=0 Nov 01 22:47:10.705342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.705195 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" Nov 01 22:47:10.705342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.705235 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" event={"ID":"54daeafc-beff-43f2-b420-e24468782220","Type":"ContainerDied","Data":"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934"} Nov 01 22:47:10.705342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.705288 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7" event={"ID":"54daeafc-beff-43f2-b420-e24468782220","Type":"ContainerDied","Data":"ae37b6c78687b72df7c41b1bc4db310bb2c3f7ca0c625b8a4cc1af540b1f6bb6"} Nov 01 22:47:10.705342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.705308 5778 scope.go:122] "RemoveContainer" containerID="30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934" Nov 01 22:47:10.720944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.720877 5778 scope.go:122] "RemoveContainer" containerID="30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934" Nov 01 22:47:10.721278 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:10.721244 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934\": container with ID starting with 30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934 not found: ID does not exist" containerID="30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934" Nov 01 22:47:10.721406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.721286 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934"} err="failed to get container status \"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934\": rpc error: code = NotFound desc = could not find container \"30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934\": container with ID starting with 30a9924f1491528f8da620b12f8d81af83f489c0dc9f0b4f97f31d88b8e83934 not found: ID does not exist" Nov 01 22:47:10.902299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902268 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.902479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902355 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.902479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902376 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.902615 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902597 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\" (UniqueName: \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.902670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902573 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:10.902670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902632 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\" (UniqueName: \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.902670 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902644 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:10.902830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.902686 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\") pod \"54daeafc-beff-43f2-b420-e24468782220\" (UID: \"54daeafc-beff-43f2-b420-e24468782220\") " Nov 01 22:47:10.903039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.903007 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:10.903386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.903367 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:10.903458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.903393 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:10.903458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.903407 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:10.905624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.905592 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "kube-api-access-qgvzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:10.905624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.905618 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:10.905976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:10.905957 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm" pod "54daeafc-beff-43f2-b420-e24468782220" (UID: "54daeafc-beff-43f2-b420-e24468782220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:11.004763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:11.004732 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54daeafc-beff-43f2-b420-e24468782220-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.004763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:11.004760 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvzn\" (UniqueName: \"kubernetes.io/projected/54daeafc-beff-43f2-b420-e24468782220-kube-api-access-qgvzn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.005015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:11.004789 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54daeafc-beff-43f2-b420-e24468782220-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.032101 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:11.032069 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:47:11.037551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:11.037517 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-30578128/e2e-lifecycle-model-30578128-kserve-986686bd5-cqqg7"] Nov 01 22:47:12.375494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:12.375458 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54daeafc-beff-43f2-b420-e24468782220" path="/var/lib/kubelet/pods/54daeafc-beff-43f2-b420-e24468782220/volumes" Nov 01 22:47:18.884884 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:18.884518 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:47:25.356248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.356196 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:47:25.665913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.665841 5778 cpu_manager.go:414] "containerMap: removing container" podUID="54daeafc-beff-43f2-b420-e24468782220" containerName="main" Nov 01 22:47:25.666174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.666151 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54daeafc-beff-43f2-b420-e24468782220" containerName="main" Nov 01 22:47:25.667418 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.666530 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="54daeafc-beff-43f2-b420-e24468782220" containerName="main" Nov 01 22:47:25.695750 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.695720 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:47:25.696262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.696111 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.711802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.711682 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-58c957-openshift-default-dockercfg-gtdvj\"" Nov 01 22:47:25.866876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.866698 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.866876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.866820 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.867150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.866915 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.867150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.867095 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cw6\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.867150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.867138 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.867306 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.867179 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.869228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.868722 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.869228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.868901 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.869228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.868933 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970556 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970453 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cw6\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970556 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970518 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970556 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970558 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970583 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970689 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970716 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.970910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970828 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.971098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.970934 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.971098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.971015 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.971698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.971670 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.971912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.971737 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.971992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.971687 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.972425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.972355 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.972619 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.972595 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.974298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.974274 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.974391 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.974329 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.979660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.979632 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:25.979833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:25.979816 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cw6\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\") pod \"e2e-inf-a-58c957-openshift-default-58b6557565-nv77f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:26.060648 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.060615 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:26.391618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.391594 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:47:26.394777 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.394716 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:26.395015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.394986 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:26.395172 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.395151 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:47:26.923609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.923573 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerStarted","Data":"3d123c8a102788ae4baba2b83a87d4def732ee523007dbd5de0e3d0a6920c7b9"} Nov 01 22:47:26.923818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.923634 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerStarted","Data":"ef976b098154c4cb45122b7e4da86cac1ecf5532dc2416b18fc5a8972246840b"} Nov 01 22:47:26.971157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:26.971106 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podStartSLOduration=1.9710916200000002 podStartE2EDuration="1.97109162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:26.96640621 +0000 UTC m=+1416.742393041" watchObservedRunningTime="2026-11-01 22:47:26.97109162 +0000 UTC m=+1416.747078448" Nov 01 22:47:27.061492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:27.061454 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:27.069361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:27.069326 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:27.931003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:27.930973 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:27.932444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:27.932420 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:47:41.957039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:41.957005 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:41.957592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:41.957278 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" containerID="cri-o://37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f" gracePeriod=30 Nov 01 22:47:47.345050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.345022 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:47.400292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400260 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400373 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400446 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400522 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400563 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400592 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400621 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400720 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.400932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.400801 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\") pod \"8d8d2275-d192-4196-8a07-711f8db86ef3\" (UID: \"8d8d2275-d192-4196-8a07-711f8db86ef3\") " Nov 01 22:47:47.401507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.401478 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.401729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.401631 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.401729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.401719 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.401883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.401754 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:47.402007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.401934 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.402142 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.402120 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.402148 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8d2275-d192-4196-8a07-711f8db86ef3-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.402164 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.402179 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.402193 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.403606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.403579 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "kube-api-access-99fs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:47.404193 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.404156 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:47.404419 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.404387 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:47.405340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.405304 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy" pod "8d8d2275-d192-4196-8a07-711f8db86ef3" (UID: "8d8d2275-d192-4196-8a07-711f8db86ef3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.463982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.463950 5778 generic.go:356] "Generic (PLEG): container finished" podID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerID="37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f" exitCode=0 Nov 01 22:47:47.464186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.463987 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" event={"ID":"8d8d2275-d192-4196-8a07-711f8db86ef3","Type":"ContainerDied","Data":"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f"} Nov 01 22:47:47.464186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.464065 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" event={"ID":"8d8d2275-d192-4196-8a07-711f8db86ef3","Type":"ContainerDied","Data":"1593a542d4e3c501f111dedb1693fe0d52e9cc08384127aac732549c87045b57"} Nov 01 22:47:47.464186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.464003 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh" Nov 01 22:47:47.464186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.464096 5778 scope.go:122] "RemoveContainer" containerID="37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f" Nov 01 22:47:47.477035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.477008 5778 scope.go:122] "RemoveContainer" containerID="37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f" Nov 01 22:47:47.477330 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:47.477309 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f\": container with ID starting with 37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f not found: ID does not exist" containerID="37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f" Nov 01 22:47:47.477417 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.477343 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f"} err="failed to get container status \"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f\": rpc error: code = NotFound desc = could not find container \"37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f\": container with ID starting with 37c86dec494e5a5d648652e66bbd51a823edf4c3911ff80468c8be62b0b77f6f not found: ID does not exist" Nov 01 22:47:47.495267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.495237 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:47.502712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.502685 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-tgnxh"] Nov 01 22:47:47.503551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.503533 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fs4\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-kube-api-access-99fs4\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.503620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.503554 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.503620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.503565 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.503620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:47.503574 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8d2275-d192-4196-8a07-711f8db86ef3-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:48.375306 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:48.375274 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" path="/var/lib/kubelet/pods/8d8d2275-d192-4196-8a07-711f8db86ef3/volumes" Nov 01 22:47:48.775466 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:48.775426 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:47:51.128876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.124906 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:47:51.128876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.126024 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" Nov 01 22:47:51.128876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.126044 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" Nov 01 22:47:51.128876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.126249 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8d2275-d192-4196-8a07-711f8db86ef3" containerName="istio-proxy" Nov 01 22:47:51.150901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.144006 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.150901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.150474 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:47:51.154873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.154066 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-58c957-serving-cert\"" Nov 01 22:47:51.308071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.308030 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.308236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.308093 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.308375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.308346 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.308493 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.308397 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnns\" (UniqueName: \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.411946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.411911 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.412166 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.412039 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnns\" (UniqueName: \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.412246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.412167 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.412302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.412242 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.412789 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:51.412747 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-serving-cert: secret "maas-api-e2e-inf-a-58c957-serving-cert" not found Nov 01 22:47:51.413018 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:51.413002 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls podName:0f58e093-c761-4238-9205-39f663f29985 nodeName:}" failed. No retries permitted until 2026-11-01 22:47:51.912979125 +0000 UTC m=+1441.688965933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls") pod "maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" (UID: "0f58e093-c761-4238-9205-39f663f29985") : secret "maas-api-e2e-inf-a-58c957-serving-cert" not found Nov 01 22:47:51.413494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.413380 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.435922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.429809 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnns\" (UniqueName: \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.435922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.430183 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.925927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.924162 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:51.963873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:51.952470 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") pod \"maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:52.158878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:52.158231 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:52.537029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:52.533652 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:47:53.615935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:53.615897 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" event={"ID":"0f58e093-c761-4238-9205-39f663f29985","Type":"ContainerStarted","Data":"290db7ed06e29d6e41590d17771fc358bd38cfb014d1bc377287350bca09ff8c"} Nov 01 22:47:53.616444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:53.616045 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" event={"ID":"0f58e093-c761-4238-9205-39f663f29985","Type":"ContainerStarted","Data":"55dee19c7fe41e61d9cdf8fd811fc8dab0549c113b4f654872b2ff1285e83846"} Nov 01 22:47:53.616444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:53.616179 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:53.640949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:53.640897 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" podStartSLOduration=2.183832759 podStartE2EDuration="2.640881249s" totalImagesPullingTime="457.04849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:53.63637224 +0000 UTC m=+1443.412359070" watchObservedRunningTime="2026-11-01 22:47:53.640881249 +0000 UTC m=+1443.416868076" Nov 01 22:47:57.258504 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.258470 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:47:57.265349 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.265324 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.278819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.278788 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1799f8\"/\"default-dockercfg-cp4rt\"" Nov 01 22:47:57.279001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.278906 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1799f8\"/\"kube-root-ca.crt\"" Nov 01 22:47:57.279001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.278984 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1799f8\"/\"test-model-1799f8-kserve-self-signed-certs\"" Nov 01 22:47:57.279135 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.279045 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1799f8\"/\"openshift-service-ca.crt\"" Nov 01 22:47:57.279649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.279622 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:47:57.379917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.379873 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.380126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.379948 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.380126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.379995 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.380126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.380089 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdd7\" (UniqueName: \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.380645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.380314 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.380645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.380507 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.481932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.481892 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482012 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482058 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482102 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482186 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdd7\" (UniqueName: \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482295 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.482786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.482761 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.483260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.483219 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.483653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.483624 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.527959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.485307 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.527959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.486999 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.607877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.586592 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdd7\" (UniqueName: \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\") pod \"test-model-1799f8-kserve-5d58c8c895-skvqc\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:57.607877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:57.586943 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:58.205186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:58.205160 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:47:58.205905 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:47:58.205874 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fcb0993_b3bc_4c5c_b521_13be98647bcb.slice/crio-b08b2f3ca21579e2676a82716561785f587f728be4c7f0333970d78a3217d81b WatchSource:0}: Error finding container b08b2f3ca21579e2676a82716561785f587f728be4c7f0333970d78a3217d81b: Status 404 returned error can't find the container with id b08b2f3ca21579e2676a82716561785f587f728be4c7f0333970d78a3217d81b Nov 01 22:47:58.731522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:58.731483 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" event={"ID":"4fcb0993-b3bc-4c5c-b521-13be98647bcb","Type":"ContainerStarted","Data":"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29"} Nov 01 22:47:58.732078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:58.731537 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" event={"ID":"4fcb0993-b3bc-4c5c-b521-13be98647bcb","Type":"ContainerStarted","Data":"b08b2f3ca21579e2676a82716561785f587f728be4c7f0333970d78a3217d81b"} Nov 01 22:47:58.784276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:58.784234 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podStartSLOduration=1.7842208149999998 podStartE2EDuration="1.784220815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:58.77714435 +0000 UTC m=+1448.553131179" watchObservedRunningTime="2026-11-01 22:47:58.784220815 +0000 UTC m=+1448.560207643" Nov 01 22:47:59.425404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.425371 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:47:59.436908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.434301 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.462053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.462017 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e36490-serving-cert\"" Nov 01 22:47:59.464373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.464273 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:47:59.568345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.568303 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.568547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.568385 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6jb\" (UniqueName: \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.568547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.568427 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.568547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.568508 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.590159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.590126 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:47:59.656926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.655403 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.670368 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6jb\" (UniqueName: \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.670431 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.670514 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.670827 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:59.671079 5778 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-serving-cert: secret "maas-api-e2e-api-b-e36490-serving-cert" not found Nov 01 22:47:59.673878 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:47:59.671143 5778 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls podName:9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f nodeName:}" failed. No retries permitted until 2026-11-01 22:48:00.171123077 +0000 UTC m=+1449.947109887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls") pod "maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" (UID: "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f") : secret "maas-api-e2e-api-b-e36490-serving-cert" not found Nov 01 22:47:59.675737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.675601 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.678201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.678175 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:47:59.681249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:47:59.681225 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6jb\" (UniqueName: \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:00.194881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:00.189085 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:00.207915 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:00.205503 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") pod \"maas-api-e2e-api-b-e36490-77bfc7d997-m67r7\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:00.381760 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:00.381727 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:00.698630 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:48:00.698575 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd48ea2_0a2c_40ef_8d0f_99b8b5ed550f.slice/crio-3f212e0c2559c129a2a1557cd167360453465cc356490fef43e67aa8380e6b0e WatchSource:0}: Error finding container 3f212e0c2559c129a2a1557cd167360453465cc356490fef43e67aa8380e6b0e: Status 404 returned error can't find the container with id 3f212e0c2559c129a2a1557cd167360453465cc356490fef43e67aa8380e6b0e Nov 01 22:48:00.699319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:00.699286 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:48:00.751401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:00.751303 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" event={"ID":"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f","Type":"ContainerStarted","Data":"3f212e0c2559c129a2a1557cd167360453465cc356490fef43e67aa8380e6b0e"} Nov 01 22:48:01.762544 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:01.762506 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" event={"ID":"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f","Type":"ContainerStarted","Data":"8aae78c096537374bc9ec3319e728fa306f25d15e7ee6ab6d5d6a6275fa5038b"} Nov 01 22:48:01.763076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:01.762585 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:01.782662 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:01.782606 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" podStartSLOduration=2.326358818 podStartE2EDuration="2.782591531s" totalImagesPullingTime="456.232713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:01.78150476 +0000 UTC m=+1451.557491589" watchObservedRunningTime="2026-11-01 22:48:01.782591531 +0000 UTC m=+1451.558578404" Nov 01 22:48:07.589278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:07.589238 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:48:07.609889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:07.609844 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:48:07.778589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:07.778559 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:48:07.876743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:07.876662 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:48:12.737192 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:12.737151 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:48:12.737788 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:12.737410 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" podUID="2f007021-12ac-4dab-bce9-758ad3556e6e" containerName="maas-api" containerID="cri-o://730f32a7163efdcd7d77ff5baaf4a278ce89170252cf2ccdc054e821ae8e24f7" gracePeriod=30 Nov 01 22:48:13.075935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.072973 5778 generic.go:356] "Generic (PLEG): container finished" podID="2f007021-12ac-4dab-bce9-758ad3556e6e" containerID="730f32a7163efdcd7d77ff5baaf4a278ce89170252cf2ccdc054e821ae8e24f7" exitCode=0 Nov 01 22:48:13.075935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.073164 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" event={"ID":"2f007021-12ac-4dab-bce9-758ad3556e6e","Type":"ContainerDied","Data":"730f32a7163efdcd7d77ff5baaf4a278ce89170252cf2ccdc054e821ae8e24f7"} Nov 01 22:48:13.678719 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.678689 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:48:13.848786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.848679 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\") pod \"2f007021-12ac-4dab-bce9-758ad3556e6e\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " Nov 01 22:48:13.848786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.848787 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") pod \"2f007021-12ac-4dab-bce9-758ad3556e6e\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " Nov 01 22:48:13.849449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.848897 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\" (UniqueName: \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\") pod \"2f007021-12ac-4dab-bce9-758ad3556e6e\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " Nov 01 22:48:13.849449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.849035 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\") pod \"2f007021-12ac-4dab-bce9-758ad3556e6e\" (UID: \"2f007021-12ac-4dab-bce9-758ad3556e6e\") " Nov 01 22:48:13.849449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.849160 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca" pod "2f007021-12ac-4dab-bce9-758ad3556e6e" (UID: "2f007021-12ac-4dab-bce9-758ad3556e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:13.850321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.850292 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f007021-12ac-4dab-bce9-758ad3556e6e-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:13.851478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.851419 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs" pod "2f007021-12ac-4dab-bce9-758ad3556e6e" (UID: "2f007021-12ac-4dab-bce9-758ad3556e6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:13.851573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.851482 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls" pod "2f007021-12ac-4dab-bce9-758ad3556e6e" (UID: "2f007021-12ac-4dab-bce9-758ad3556e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:13.851573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.851563 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt" pod "2f007021-12ac-4dab-bce9-758ad3556e6e" (UID: "2f007021-12ac-4dab-bce9-758ad3556e6e"). InnerVolumeSpecName "kube-api-access-nrrdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:13.951609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.951570 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrdt\" (UniqueName: \"kubernetes.io/projected/2f007021-12ac-4dab-bce9-758ad3556e6e-kube-api-access-nrrdt\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:13.951609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.951606 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:13.951609 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:13.951616 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f007021-12ac-4dab-bce9-758ad3556e6e-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:14.086089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.086048 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" event={"ID":"2f007021-12ac-4dab-bce9-758ad3556e6e","Type":"ContainerDied","Data":"c2f86d96ea02dbbf661ee484ba3846c030becf2231328ac094316e69f9aaedb1"} Nov 01 22:48:14.086304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.086130 5778 scope.go:122] "RemoveContainer" containerID="730f32a7163efdcd7d77ff5baaf4a278ce89170252cf2ccdc054e821ae8e24f7" Nov 01 22:48:14.086396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.086303 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x" Nov 01 22:48:14.128012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.127976 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:48:14.131359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.131330 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2476f8-7bcdd6f696-7jk7x"] Nov 01 22:48:14.376910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.376808 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f007021-12ac-4dab-bce9-758ad3556e6e" path="/var/lib/kubelet/pods/2f007021-12ac-4dab-bce9-758ad3556e6e/volumes" Nov 01 22:48:14.828583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.828545 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:48:14.830253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.830224 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2f007021-12ac-4dab-bce9-758ad3556e6e" containerName="maas-api" Nov 01 22:48:14.830253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.830254 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f007021-12ac-4dab-bce9-758ad3556e6e" containerName="maas-api" Nov 01 22:48:14.830736 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.830703 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f007021-12ac-4dab-bce9-758ad3556e6e" containerName="maas-api" Nov 01 22:48:14.871425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.871389 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:48:14.872273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.871656 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.877749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.877721 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44a9aa\"/\"test-model-44a9aa-kserve-self-signed-certs\"" Nov 01 22:48:14.878361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.878339 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44a9aa\"/\"default-dockercfg-tk92r\"" Nov 01 22:48:14.878597 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.878583 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44a9aa\"/\"kube-root-ca.crt\"" Nov 01 22:48:14.878778 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.878762 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44a9aa\"/\"openshift-service-ca.crt\"" Nov 01 22:48:14.970844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.970799 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.971066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.970916 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.971066 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.970974 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.971199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.971090 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltqg\" (UniqueName: \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.971199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.971126 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:14.971199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:14.971184 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074052 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltqg\" (UniqueName: \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074117 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074178 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074397 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074479 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.075223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.074537 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.089415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.089280 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.089955 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.089931 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.093875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.090348 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.096044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.095702 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.106060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.096417 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.138001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.137810 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltqg\" (UniqueName: \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\") pod \"test-model-44a9aa-kserve-7d96b84dc-n9jm9\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.197184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.197146 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:15.626571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:15.626540 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:48:15.627352 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:48:15.627326 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d350e26_8da2_4621_b04d_190984552906.slice/crio-cd6a2dcf9cc100ad4151cdee9ecf817308aefe890c0649b20a32c405910a2caa WatchSource:0}: Error finding container cd6a2dcf9cc100ad4151cdee9ecf817308aefe890c0649b20a32c405910a2caa: Status 404 returned error can't find the container with id cd6a2dcf9cc100ad4151cdee9ecf817308aefe890c0649b20a32c405910a2caa Nov 01 22:48:16.204875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:16.198083 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" event={"ID":"9d350e26-8da2-4621-b04d-190984552906","Type":"ContainerStarted","Data":"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af"} Nov 01 22:48:16.204875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:16.198138 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" event={"ID":"9d350e26-8da2-4621-b04d-190984552906","Type":"ContainerStarted","Data":"cd6a2dcf9cc100ad4151cdee9ecf817308aefe890c0649b20a32c405910a2caa"} Nov 01 22:48:16.204875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:16.202918 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:16.253878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:16.253764 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podStartSLOduration=2.253742125 podStartE2EDuration="2.253742125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:16.237269865 +0000 UTC m=+1466.013256696" watchObservedRunningTime="2026-11-01 22:48:16.253742125 +0000 UTC m=+1466.029728956" Nov 01 22:48:18.776387 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:48:18.776348 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:48:20.942569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:20.942521 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:48:20.959732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:20.959700 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:20.965945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:20.965913 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:48:20.998463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:20.998433 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e716d-openshift-default-dockercfg-62t9z\"" Nov 01 22:48:21.094905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.085273 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.094905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.085372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105087 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105445 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105520 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105599 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105649 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmtk\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.105765 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.112888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.106110 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.222952 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223009 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223041 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223081 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223112 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmtk\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223200 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223286 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223515 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.226925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.223597 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.227516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.227264 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.227579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.227536 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.232946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.227790 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.232946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.228238 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.232946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.228458 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.232946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.228668 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.249014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.239483 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.308963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.283534 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.308963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.287690 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmtk\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\") pod \"e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:21.594930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:21.588867 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:22.072147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.072114 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:48:22.073310 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:48:22.073282 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5596ee8f_f355_4d12_bb50_aef28f0dbcf3.slice/crio-d074f488ddbfbd280003c6780d24599b7291bb9e394abecd723774a6f2afbdc5 WatchSource:0}: Error finding container d074f488ddbfbd280003c6780d24599b7291bb9e394abecd723774a6f2afbdc5: Status 404 returned error can't find the container with id d074f488ddbfbd280003c6780d24599b7291bb9e394abecd723774a6f2afbdc5 Nov 01 22:48:22.076548 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.076510 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:22.076636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.076599 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:22.076687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.076648 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:22.292467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.292025 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerStarted","Data":"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf"} Nov 01 22:48:22.292467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.292098 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerStarted","Data":"d074f488ddbfbd280003c6780d24599b7291bb9e394abecd723774a6f2afbdc5"} Nov 01 22:48:22.322343 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.322251 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podStartSLOduration=2.322219212 podStartE2EDuration="2.322219212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:22.314739266 +0000 UTC m=+1472.090726095" watchObservedRunningTime="2026-11-01 22:48:22.322219212 +0000 UTC m=+1472.098206040" Nov 01 22:48:22.589733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.589644 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:22.659410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.659367 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:48:22.696877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.696290 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.707527 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.704397 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817081 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817154 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817198 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817339 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817374 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgvs\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817402 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817442 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817477 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.821949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.817575 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.822992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.822963 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e202e7-openshift-default-dockercfg-s5jpf\"" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.918980 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919034 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919129 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919283 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919328 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919369 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919518 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919554 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgvs\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.919582 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.920072 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.920307 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.920917 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.921919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.921506 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.933822 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.926370 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.935889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.935688 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.936955 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.936932 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgvs\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.938077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.938056 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:22.942420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:22.942376 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:23.078325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.078282 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:23.398036 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.398010 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:48:23.399233 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:48:23.399203 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c3560c_cbf5_4e5a_bb54_9224c4fe1845.slice/crio-2e31089024285cfbe407a53ce22ab2d1947c3e8742c2838e440d1d7c1a30b182 WatchSource:0}: Error finding container 2e31089024285cfbe407a53ce22ab2d1947c3e8742c2838e440d1d7c1a30b182: Status 404 returned error can't find the container with id 2e31089024285cfbe407a53ce22ab2d1947c3e8742c2838e440d1d7c1a30b182 Nov 01 22:48:23.403218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.403179 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:23.403298 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.403267 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:23.403341 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.403313 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:23.602457 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:23.602423 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:24.312710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:24.312674 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" event={"ID":"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845","Type":"ContainerStarted","Data":"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3"} Nov 01 22:48:24.313272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:24.312737 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" event={"ID":"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845","Type":"ContainerStarted","Data":"2e31089024285cfbe407a53ce22ab2d1947c3e8742c2838e440d1d7c1a30b182"} Nov 01 22:48:24.313272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:24.312904 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:24.314140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:24.314124 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:48:24.342873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:24.342534 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podStartSLOduration=2.342513291 podStartE2EDuration="2.342513291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:24.33513321 +0000 UTC m=+1474.111120038" watchObservedRunningTime="2026-11-01 22:48:24.342513291 +0000 UTC m=+1474.118500121" Nov 01 22:48:25.081826 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.079188 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:25.099202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.084367 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:25.099202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.084430 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:25.198479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.198448 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:25.228877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.225359 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:25.374816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:25.374718 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:48:26.091928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:26.091505 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:26.091928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:26.091567 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:27.080300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:27.078817 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:27.080300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:27.078898 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:28.079356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:28.079322 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:28.079627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:28.079587 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:29.079660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:29.079628 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:29.080243 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:29.079680 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:30.088324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:30.088285 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:30.088985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:30.088950 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:31.079698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:31.079660 5778 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 22:48:31.079934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:31.079715 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 22:48:32.093839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:32.093804 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:32.094370 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:32.094242 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:32.095149 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:32.095127 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:48:40.864178 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:40.864135 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded" Nov 01 22:48:41.102501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:41.102435 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:48:48.775296 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:48:48.775253 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:48:50.654558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.654525 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:48:50.659077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.659050 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:48:50.773294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.773260 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:48:50.784598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.784567 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.787280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.787249 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:48:50.793342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.793318 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e36490-openshift-default-dockercfg-vjrhb\"" Nov 01 22:48:50.878511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878473 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878541 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878577 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878606 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878709 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878738 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.878921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878883 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.879017 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.878964 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.879086 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.879067 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5nz\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.980704 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.980624 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.980867 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.980758 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.980867 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.980810 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.980974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.980956 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5nz\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981086 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981069 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981123 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981161 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981201 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981254 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981426 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981438 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981478 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.981827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.981755 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.982283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.982259 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.983590 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.983567 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.984361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.984342 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.990286 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.990261 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5nz\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:50.991356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:50.991339 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:51.143000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:51.142967 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:51.456356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:51.456320 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:48:51.459949 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:48:51.459916 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e67e68_5ca5_4994_b8c3_37c3e9ba42cc.slice/crio-9858eb3e714ffc1bbe05a2c807d4fd03d6369750f68f108dabb372b40e6a73b3 WatchSource:0}: Error finding container 9858eb3e714ffc1bbe05a2c807d4fd03d6369750f68f108dabb372b40e6a73b3: Status 404 returned error can't find the container with id 9858eb3e714ffc1bbe05a2c807d4fd03d6369750f68f108dabb372b40e6a73b3 Nov 01 22:48:51.463516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:51.463476 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:51.463622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:51.463564 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:51.463622 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:51.463606 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:52.144108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:52.144070 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" event={"ID":"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc","Type":"ContainerStarted","Data":"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2"} Nov 01 22:48:52.144656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:52.144129 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" event={"ID":"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc","Type":"ContainerStarted","Data":"9858eb3e714ffc1bbe05a2c807d4fd03d6369750f68f108dabb372b40e6a73b3"} Nov 01 22:48:52.175829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:52.175770 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podStartSLOduration=2.175756513 podStartE2EDuration="2.175756513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:52.171317892 +0000 UTC m=+1501.947304731" watchObservedRunningTime="2026-11-01 22:48:52.175756513 +0000 UTC m=+1501.951743425" Nov 01 22:48:53.150000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.146917 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:48:53.150625 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.150590 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:53.150726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.150653 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:53.162591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.162514 5778 generic.go:356] "Generic (PLEG): container finished" podID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerID="3d123c8a102788ae4baba2b83a87d4def732ee523007dbd5de0e3d0a6920c7b9" exitCode=0 Nov 01 22:48:53.162733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.162591 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerDied","Data":"3d123c8a102788ae4baba2b83a87d4def732ee523007dbd5de0e3d0a6920c7b9"} Nov 01 22:48:53.163139 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.163115 5778 scope.go:122] "RemoveContainer" containerID="3d123c8a102788ae4baba2b83a87d4def732ee523007dbd5de0e3d0a6920c7b9" Nov 01 22:48:53.165081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.165054 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:53.165167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.165119 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:53.165167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:53.165150 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:54.172891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:54.154400 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:54.172891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:54.154463 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:54.201227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:54.201179 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerStarted","Data":"5ad7205a852bd3d848fd28d1ec047707e002d905b9dc88e16b1a3e3d29dbca34"} Nov 01 22:48:55.083928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:55.066908 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:48:55.083928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:55.067094 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:48:55.083928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:55.067138 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:48:55.152873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:55.147274 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:55.152873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:55.147335 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:56.064879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.064555 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:48:56.064879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.064651 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:48:56.157880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.146318 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:56.157880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.146381 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:56.704455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.704412 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:56.822870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.813700 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 22:48:56.846330 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.843760 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:56.861881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.855963 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:56.863278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.863246 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-021714-openshift-default-dockercfg-vmtmc\"" Nov 01 22:48:56.990520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:56.990381 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:57.016953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.016914 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 22:48:57.040028 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.039979 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:57.057479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.057435 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.083104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.083058 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.084277 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.084239 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.084521 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.084500 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.084770 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.084752 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.084939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.084920 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.085082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.085066 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.085254 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.085238 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.085614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.085591 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwck2\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.086060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.063838 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:48:57.086259 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.086232 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:48:57.110029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.109950 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:57.126891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.125945 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:57.149891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.145713 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:57.149891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.145773 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:57.209691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.209602 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210002 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210145 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210251 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210466 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210454 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210558 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210654 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.210864 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.210832 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.211151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.211136 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwck2\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.212329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.212307 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.214811 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.214649 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.239032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.215220 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.246927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.226149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.296225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.273384 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.346881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.329666 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.346881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.331701 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.427876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.421937 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:57.487874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.471882 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.511707 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.490270 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwck2\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.618508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.618160 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:48:57.974096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:57.950001 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:48:58.174154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.066231 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:48:58.174154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.066296 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:48:58.174154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.143579 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:58.174154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.143640 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:58.624788 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.624735 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 22:48:58.718989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.703959 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:58.718989 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.704070 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:58.798715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.791036 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:48:58.868354 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:58.863951 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerStarted","Data":"8db02463a9ade345c56bc952bb65c3e7449455efba4129a90912afe09780f669"} Nov 01 22:48:59.092882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.090483 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:48:59.092882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.090543 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:48:59.119940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.117540 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:59.152576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.147186 5778 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 22:48:59.152576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.147283 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 22:48:59.892509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.890913 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerStarted","Data":"6263b3139ea057744b3e4e1d5371cd59f33b68874942cf9abf3d2d266f75d191"} Nov 01 22:48:59.927906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:48:59.926050 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podStartSLOduration=3.926030503 podStartE2EDuration="3.926030503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:59.918391854 +0000 UTC m=+1509.694378681" watchObservedRunningTime="2026-11-01 22:48:59.926030503 +0000 UTC m=+1509.702017333" Nov 01 22:49:00.074876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.065271 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:49:00.074876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.065334 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:49:00.085022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.083009 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:49:00.123884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.111097 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.123884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.120715 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e202e7\"/\"kube-root-ca.crt\"" Nov 01 22:49:00.128967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.124628 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e202e7\"/\"test-model-e202e7-kserve-self-signed-certs\"" Nov 01 22:49:00.128967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.124806 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e202e7\"/\"default-dockercfg-2xczs\"" Nov 01 22:49:00.128967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.125002 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e202e7\"/\"openshift-service-ca.crt\"" Nov 01 22:49:00.181983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.160208 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:49:00.198520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.197144 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:49:00.200757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.200726 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:49:00.208803 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.208769 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.213744 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.213802 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.213921 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlhk\" (UniqueName: \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.213969 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.214127 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.214440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.214234 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.321790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.321746 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.322274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.322245 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.322463 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.322444 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.322745 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.322709 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlhk\" (UniqueName: \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.322931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.322913 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.323501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.323482 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.325238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.325213 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.329876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.325831 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.342724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.331067 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.607453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.586635 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.621911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.621655 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.623074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.622380 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlhk\" (UniqueName: \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\") pod \"test-model-e202e7-kserve-c96679b4f-tf2sf\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:00.629209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.624897 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:49:00.629209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.624947 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:49:00.629209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.625790 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:49:00.809504 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:00.809013 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:01.197723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.197685 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:49:01.198394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.198355 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 22:49:01.479408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.479368 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:49:01.479918 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:49:01.479829 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeab12b5d_fabd_4ac3_a300_f735dcc791ff.slice/crio-ef81ab40034f582aec545b2588cc96f480d8ff4d8d99bd352d9b67d30de33e5d WatchSource:0}: Error finding container ef81ab40034f582aec545b2588cc96f480d8ff4d8d99bd352d9b67d30de33e5d: Status 404 returned error can't find the container with id ef81ab40034f582aec545b2588cc96f480d8ff4d8d99bd352d9b67d30de33e5d Nov 01 22:49:01.638571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.638198 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:49:01.918602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.918565 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" event={"ID":"eab12b5d-fabd-4ac3-a300-f735dcc791ff","Type":"ContainerStarted","Data":"2a6a03e056f1236383b9cb14dfa4e271c84a4ae97a34658daae10f3808d00b1b"} Nov 01 22:49:01.918602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.918616 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" event={"ID":"eab12b5d-fabd-4ac3-a300-f735dcc791ff","Type":"ContainerStarted","Data":"ef81ab40034f582aec545b2588cc96f480d8ff4d8d99bd352d9b67d30de33e5d"} Nov 01 22:49:01.919928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.919357 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:49:01.920493 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.920473 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:49:01.976304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:01.976231 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podStartSLOduration=2.976207383 podStartE2EDuration="2.976207383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:01.942877532 +0000 UTC m=+1511.718864362" watchObservedRunningTime="2026-11-01 22:49:01.976207383 +0000 UTC m=+1511.752194244" Nov 01 22:49:02.265869 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:02.264265 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:49:02.266344 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:02.266007 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:49:02.809809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:02.809770 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:10.810250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:10.810215 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:10.825325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:10.825290 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:11.144977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:11.144900 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:18.159453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.155932 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:49:18.166207 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.166119 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:49:18.166371 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.166241 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.175215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.174864 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f07902\"/\"openshift-service-ca.crt\"" Nov 01 22:49:18.175215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.174920 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f07902\"/\"kube-root-ca.crt\"" Nov 01 22:49:18.175996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.175972 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f07902\"/\"default-dockercfg-rl4c5\"" Nov 01 22:49:18.176132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.176103 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f07902\"/\"test-model-f07902-kserve-self-signed-certs\"" Nov 01 22:49:18.185600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.185573 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.185749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.185631 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.185749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.185730 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.185912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.185793 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.186061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.186025 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.186818 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.186788 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9kc\" (UniqueName: \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.288942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.288903 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.288967 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.289017 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.289077 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.289182 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.289266 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9kc\" (UniqueName: \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.289985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.289955 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.290170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.290146 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.290306 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.290045 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.292741 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.292716 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.293438 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.293418 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.302423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.302397 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9kc\" (UniqueName: \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\") pod \"test-model-f07902-kserve-7547dddd6c-qw7gd\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.529957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:18.521770 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:18.804010 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:49:18.803899 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:49:19.001255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:19.001224 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:49:19.001980 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:49:19.001948 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d9b8603_13ce_4f85_9737_83efaeed82a2.slice/crio-9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9 WatchSource:0}: Error finding container 9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9: Status 404 returned error can't find the container with id 9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9 Nov 01 22:49:19.654221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:19.654156 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" event={"ID":"2d9b8603-13ce-4f85-9737-83efaeed82a2","Type":"ContainerStarted","Data":"76d868d85de89c8a700703a358f5c4a372d1c8e751d60a9834126e7c36054661"} Nov 01 22:49:19.654221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:19.654217 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" event={"ID":"2d9b8603-13ce-4f85-9737-83efaeed82a2","Type":"ContainerStarted","Data":"9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9"} Nov 01 22:49:19.681230 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:19.681105 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podStartSLOduration=1.681088782 podStartE2EDuration="1.681088782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:19.679026235 +0000 UTC m=+1529.455013061" watchObservedRunningTime="2026-11-01 22:49:19.681088782 +0000 UTC m=+1529.457075660" Nov 01 22:49:20.103766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.103669 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:49:20.108715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.108686 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.121874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.119427 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-58c957\"/\"default-dockercfg-dxmsb\"" Nov 01 22:49:20.121874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.119558 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-58c957\"/\"test-model-58c957-kserve-self-signed-certs\"" Nov 01 22:49:20.122464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.122437 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-58c957\"/\"openshift-service-ca.crt\"" Nov 01 22:49:20.124063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.124005 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-58c957\"/\"kube-root-ca.crt\"" Nov 01 22:49:20.128124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.128099 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:49:20.252911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.216974 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.269098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.269060 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.279564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.279523 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.301874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.279943 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.302203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.302169 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdb7g\" (UniqueName: \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.317064 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.317027 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.459683 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.459802 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.459991 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.460163 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.460226 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdb7g\" (UniqueName: \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.460323 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.461007 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.478894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.461791 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.513875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.491540 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.513875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.491959 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.526014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.525979 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:20.542899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.542833 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.557950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.555244 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdb7g\" (UniqueName: \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\") pod \"test-model-58c957-kserve-64d6df7dd5-q42ll\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:20.729180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:20.729086 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:21.243881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.242470 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:49:21.527004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.525044 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:49:21.527004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.526208 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.625894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.601344 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e716d\"/\"openshift-service-ca.crt\"" Nov 01 22:49:21.625894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.601894 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e716d\"/\"kube-root-ca.crt\"" Nov 01 22:49:21.625894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.608200 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e716d\"/\"test-model-0e716d-kserve-self-signed-certs\"" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.747880 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.747931 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kxm\" (UniqueName: \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.748031 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.748107 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.748237 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.749985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.748350 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.822384 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.822148 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e716d\"/\"default-dockercfg-kmvdb\"" Nov 01 22:49:21.850386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850351 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.850568 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850399 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kxm\" (UniqueName: \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.850568 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850471 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.850568 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850519 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.850743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850619 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.850743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.850716 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.851541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.851515 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.853761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.853734 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.860758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.860220 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.869879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.866611 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.895015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.882547 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:21.895015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:21.886687 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kxm\" (UniqueName: \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\") pod \"test-model-0e716d-kserve-8649848597-jzdmt\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:22.011158 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.011114 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:49:22.011519 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:49:22.011496 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42389da7_468d_4a61_9942_c67def02d4d3.slice/crio-a5321a3a97e9917654175c9bde6019e2605345c5560eb9e2c1fe67295234d990 WatchSource:0}: Error finding container a5321a3a97e9917654175c9bde6019e2605345c5560eb9e2c1fe67295234d990: Status 404 returned error can't find the container with id a5321a3a97e9917654175c9bde6019e2605345c5560eb9e2c1fe67295234d990 Nov 01 22:49:22.036681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.036652 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:22.414176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.414153 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:49:22.791696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.791648 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" event={"ID":"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5","Type":"ContainerStarted","Data":"1d58fc300c10e42bde03c18470772a65361db1883ace110887e4c16d4f467c32"} Nov 01 22:49:22.791696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.791701 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" event={"ID":"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5","Type":"ContainerStarted","Data":"7e6442a732e9c5d7116072eb30b066d1a4b7a8db3b2116720d2895c48a6d8e5e"} Nov 01 22:49:22.793698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.793673 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" event={"ID":"42389da7-468d-4a61-9942-c67def02d4d3","Type":"ContainerStarted","Data":"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a"} Nov 01 22:49:22.793698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.793706 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" event={"ID":"42389da7-468d-4a61-9942-c67def02d4d3","Type":"ContainerStarted","Data":"a5321a3a97e9917654175c9bde6019e2605345c5560eb9e2c1fe67295234d990"} Nov 01 22:49:22.818751 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.818693 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podStartSLOduration=1.8186790369999999 podStartE2EDuration="1.818679037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:22.815736508 +0000 UTC m=+1532.591723336" watchObservedRunningTime="2026-11-01 22:49:22.818679037 +0000 UTC m=+1532.594665865" Nov 01 22:49:22.845424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:22.845368 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podStartSLOduration=2.845350888 podStartE2EDuration="2.845350888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:22.839842254 +0000 UTC m=+1532.615829118" watchObservedRunningTime="2026-11-01 22:49:22.845350888 +0000 UTC m=+1532.621337716" Nov 01 22:49:23.037287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:23.037242 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:23.729559 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:23.729524 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:28.532877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:28.528348 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:28.571402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:28.571371 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:28.960356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:28.960322 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:30.729819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:30.729714 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:30.746586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:30.746554 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:31.258926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:31.255577 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:32.037596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:32.037556 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:32.056630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:32.056600 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:32.260997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:32.260966 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:37.472548 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:37.472477 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.706889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:37.706169 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.718878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:37.717943 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.949522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:37.949478 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.000128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.000067 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.038995 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.038947 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:49:38.086115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.086039 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:49:38.086941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.086905 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.117087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.117035 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:49:38.144258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.144150 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": context deadline exceeded" Nov 01 22:49:38.242049 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.241944 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": context deadline exceeded" Nov 01 22:49:38.383062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.381981 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded" Nov 01 22:49:38.480873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.480194 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.509188 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.509090 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:49:38.594636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.594308 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": context deadline exceeded" Nov 01 22:49:38.722935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.718216 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:49:38.722935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.718792 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:49:38.866233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:38.865073 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.002114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.000933 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.002114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.001502 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:49:39.190924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.150771 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.190924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.151418 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:49:39.190924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.164047 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.190924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.165229 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:39.190924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.171531 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:49:39.208878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.191425 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.208878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.191825 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:49:39.520956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.504939 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.568340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.566096 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.630881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.392400 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.746873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.746823 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:49:39.892009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.873434 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.901501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.901471 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:49:39.958914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.958153 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.978835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:39.978774 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.018216 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.018174 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:49:40.174936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.154456 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.174936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.173968 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.175944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.175907 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": context deadline exceeded" Nov 01 22:49:40.195567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.195523 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.269908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.264991 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.282878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.270193 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.129:8000/ready\": context deadline exceeded" Nov 01 22:49:40.550549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.549925 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 22:49:40.756090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.756035 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.886037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.885117 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:49:40.886037 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.885182 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.906873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.902150 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.977019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.976957 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded" Nov 01 22:49:40.977795 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.977768 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:41.021948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.992778 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:49:41.023361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:40.998947 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:49:41.023605 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.023575 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.036873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.024270 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" containerID="cri-o://7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be" gracePeriod=30 Nov 01 22:49:41.037190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.025962 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.180202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.180148 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.183042 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.183002 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 22:49:41.210874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.183954 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:49:41.225878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.216198 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:49:41.225878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.217468 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:49:41.571371 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.545895 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.575528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.544981 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.129:8000/ready\": context deadline exceeded" Nov 01 22:49:41.576174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.576142 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.589669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.589641 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:41.589999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.589982 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:41.768780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.768738 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.842414 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.842313 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/health\": context deadline exceeded" Nov 01 22:49:41.851110 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.851066 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.868096 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.859923 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 22:49:41.890471 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.890443 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:49:41.908734 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.908685 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.965706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.965677 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:49:41.979328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:41.979227 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.055603 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.055557 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.240879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.210453 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.240879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.211206 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": context deadline exceeded" Nov 01 22:49:42.240879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.218381 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 22:49:42.548967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.547395 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.653585 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.653535 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.659728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.659671 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.129:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.697227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.697195 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:49:42.697506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.697489 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:49:42.697642 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.697630 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:49:42.697763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.697750 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:49:42.697951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.697935 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:49:42.698105 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.698091 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:49:42.698243 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:42.698232 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:49:46.436115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.436056 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.547545 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.547621 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.547692 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.547819 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.547914 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.548115 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.548169 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.548224 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.548944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.548313 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\") pod \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\" (UID: \"03acb1a5-09b5-4b7d-b213-69eb1beb342e\") " Nov 01 22:49:46.553700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.551822 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:46.553700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.553036 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:46.553700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.553257 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:46.553700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.553605 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:46.557089 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.557003 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:46.566456 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.566405 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:46.570303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.570240 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "kube-api-access-ftb2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:46.570792 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.570756 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:46.576374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.576333 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy" pod "03acb1a5-09b5-4b7d-b213-69eb1beb342e" (UID: "03acb1a5-09b5-4b7d-b213-69eb1beb342e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708569 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708604 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708620 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708634 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708648 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftb2s\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-kube-api-access-ftb2s\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708664 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708677 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708691 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.708709 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03acb1a5-09b5-4b7d-b213-69eb1beb342e-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:46.802395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.802360 5778 generic.go:356] "Generic (PLEG): container finished" podID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerID="7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be" exitCode=0 Nov 01 22:49:46.802589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.802440 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" Nov 01 22:49:46.802589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.802439 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" event={"ID":"03acb1a5-09b5-4b7d-b213-69eb1beb342e","Type":"ContainerDied","Data":"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be"} Nov 01 22:49:46.802589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.802530 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt" event={"ID":"03acb1a5-09b5-4b7d-b213-69eb1beb342e","Type":"ContainerDied","Data":"b95b137369c41d51a6f62b2d030d17de54b2f482e467ce4af3456175199afe9f"} Nov 01 22:49:46.802589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.802555 5778 scope.go:122] "RemoveContainer" containerID="7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be" Nov 01 22:49:46.817269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.816660 5778 scope.go:122] "RemoveContainer" containerID="7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be" Nov 01 22:49:46.817269 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:49:46.817087 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be\": container with ID starting with 7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be not found: ID does not exist" containerID="7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be" Nov 01 22:49:46.817269 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.817124 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be"} err="failed to get container status \"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be\": rpc error: code = NotFound desc = could not find container \"7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be\": container with ID starting with 7e8c4ab09302d52c073b782e3958fbcdf0bb8186234e6f3aaa009f6d594794be not found: ID does not exist" Nov 01 22:49:46.851870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.851820 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:49:46.869046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:46.868997 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2zmkt"] Nov 01 22:49:48.378199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:48.378166 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" path="/var/lib/kubelet/pods/03acb1a5-09b5-4b7d-b213-69eb1beb342e/volumes" Nov 01 22:49:48.782774 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:49:48.782724 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:49:49.856102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.856065 5778 generic.go:356] "Generic (PLEG): container finished" podID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerID="e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf" exitCode=0 Nov 01 22:49:49.856623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.856155 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerDied","Data":"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf"} Nov 01 22:49:49.857225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.857199 5778 scope.go:122] "RemoveContainer" containerID="e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf" Nov 01 22:49:49.860249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.860215 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:49.860352 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.860315 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:49.860401 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:49.860364 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:50.867808 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:50.867712 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerStarted","Data":"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83"} Nov 01 22:49:51.590398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:51.590365 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:49:51.591434 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:51.591406 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:51.591573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:51.591462 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:52.590186 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:52.590153 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:52.590693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:52.590209 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:53.392596 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.392507 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:49:53.393435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.393413 5778 cpu_manager.go:414] "containerMap: removing container" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" Nov 01 22:49:53.393435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.393437 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" Nov 01 22:49:53.393674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.393659 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="03acb1a5-09b5-4b7d-b213-69eb1beb342e" containerName="istio-proxy" Nov 01 22:49:53.399160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.399137 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.410884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.410829 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b78bd-openshift-default-dockercfg-hv5s7\"" Nov 01 22:49:53.468203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.468165 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddst\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.468522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.468500 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.468633 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.468619 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.468819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.468805 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.469086 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.469070 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.469197 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.469183 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.469322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.469309 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.472918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.472895 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.473139 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.473125 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.485005 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.484969 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:49:53.574710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.574651 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.574710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.574718 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.574765 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.574869 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.575000 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575175 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.575074 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddst\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.575183 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575227 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.575208 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.575369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.575355 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.576088 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.576057 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.576228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.576124 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.576228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.576150 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.576424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.576258 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.576813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.576789 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.578539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.578517 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.578669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.578650 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.585157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.585132 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddst\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.586209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.586186 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:53.589879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.589511 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:53.589879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.589562 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:53.747880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:53.743435 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:54.181722 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:49:54.181685 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b4df03_ee7f_4182_9e99_c8c45dcffba8.slice/crio-39c411b9797dfce6ff5fcab4f993bc8901f163905c2f22036efa5ebfcd5c3878 WatchSource:0}: Error finding container 39c411b9797dfce6ff5fcab4f993bc8901f163905c2f22036efa5ebfcd5c3878: Status 404 returned error can't find the container with id 39c411b9797dfce6ff5fcab4f993bc8901f163905c2f22036efa5ebfcd5c3878 Nov 01 22:49:54.182607 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.182192 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:49:54.186165 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.186126 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:54.186581 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.186555 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:54.186735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.186717 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:49:54.313736 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.313712 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:49:54.709506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.709473 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:54.709810 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:54.709770 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:55.086942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.083893 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" event={"ID":"15b4df03-ee7f-4182-9e99-c8c45dcffba8","Type":"ContainerStarted","Data":"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a"} Nov 01 22:49:55.086942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.083952 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" event={"ID":"15b4df03-ee7f-4182-9e99-c8c45dcffba8","Type":"ContainerStarted","Data":"39c411b9797dfce6ff5fcab4f993bc8901f163905c2f22036efa5ebfcd5c3878"} Nov 01 22:49:55.114387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.114337 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podStartSLOduration=2.114322497 podStartE2EDuration="2.114322497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:55.112384516 +0000 UTC m=+1564.888371346" watchObservedRunningTime="2026-11-01 22:49:55.114322497 +0000 UTC m=+1564.890309326" Nov 01 22:49:55.590263 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.590222 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:55.590446 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.590284 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:55.746626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.746587 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:49:55.759502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.759143 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:49:55.759502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:55.759205 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:49:56.594980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:56.592193 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.594980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:56.592254 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 22:49:56.781024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:56.752080 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.781024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:56.752140 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:49:57.688244 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:57.688171 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:49:57.690513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:57.690486 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:49:57.751882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:57.749309 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:49:57.751882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:57.749366 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:49:58.745825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:58.745293 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:49:58.745825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:58.745363 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:49:59.747880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:59.747269 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:49:59.747880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:49:59.747349 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:50:00.745264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:00.745224 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:50:00.745598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:00.745564 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:50:01.745937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:01.745901 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 01 22:50:01.746661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:01.746627 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 01 22:50:02.752334 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:02.752301 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:50:02.753973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:02.753953 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:50:02.754942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:02.754921 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:50:09.575905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:09.575863 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:50:09.576340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:09.576143 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" podUID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" containerName="maas-api" containerID="cri-o://8aae78c096537374bc9ec3319e728fa306f25d15e7ee6ab6d5d6a6275fa5038b" gracePeriod=30 Nov 01 22:50:10.443876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:10.442345 5778 generic.go:356] "Generic (PLEG): container finished" podID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" containerID="8aae78c096537374bc9ec3319e728fa306f25d15e7ee6ab6d5d6a6275fa5038b" exitCode=0 Nov 01 22:50:10.514392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:10.514342 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" event={"ID":"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f","Type":"ContainerDied","Data":"8aae78c096537374bc9ec3319e728fa306f25d15e7ee6ab6d5d6a6275fa5038b"} Nov 01 22:50:11.370566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.370537 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:50:11.407998 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.407958 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:50:11.408735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.408717 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" containerName="maas-api" Nov 01 22:50:11.408735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.408735 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" containerName="maas-api" Nov 01 22:50:11.408957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.408914 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" containerName="maas-api" Nov 01 22:50:11.412657 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.412633 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.435938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.425047 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:50:11.460638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.460593 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" event={"ID":"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f","Type":"ContainerDied","Data":"3f212e0c2559c129a2a1557cd167360453465cc356490fef43e67aa8380e6b0e"} Nov 01 22:50:11.460638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.460616 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7" Nov 01 22:50:11.460941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.460654 5778 scope.go:122] "RemoveContainer" containerID="8aae78c096537374bc9ec3319e728fa306f25d15e7ee6ab6d5d6a6275fa5038b" Nov 01 22:50:11.493828 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.493793 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\") pod \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " Nov 01 22:50:11.494042 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.493950 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") pod \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " Nov 01 22:50:11.494042 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494002 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\" (UniqueName: \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\") pod \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " Nov 01 22:50:11.494332 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494307 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\") pod \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\" (UID: \"9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f\") " Nov 01 22:50:11.495408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494641 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b6f\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494692 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494837 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494926 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.494976 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca" pod "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" (UID: "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:11.495735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.495570 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495794 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.495731 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.495791 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.495825 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.495997 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.495973 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.497909 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.496189 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:11.497909 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.496593 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs" pod "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" (UID: "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:11.497909 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.496824 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls" pod "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" (UID: "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:11.497909 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.496950 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb" pod "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" (UID: "9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f"). InnerVolumeSpecName "kube-api-access-xc6jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:11.597758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597663 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.597758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597724 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.597758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597745 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597821 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597947 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b6f\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598119 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.597988 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598123 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598179 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598230 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598511 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598349 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.598654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598565 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:11.598742 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598723 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:11.598890 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.598746 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6jb\" (UniqueName: \"kubernetes.io/projected/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f-kube-api-access-xc6jb\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:11.599044 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.599010 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.600315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.599295 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.600315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.599465 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.600315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.599595 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.601071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.601047 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.601840 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.601795 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.623212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.623182 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.624130 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.624107 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b6f\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.742917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.737240 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:11.884007 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.883905 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:50:11.889594 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:11.889560 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e36490-77bfc7d997-m67r7"] Nov 01 22:50:12.136670 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:50:12.136577 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860f831e_caa5_4dca_8fe8_a1fed6bdf1d3.slice/crio-eba4982a02d2008c9b6102e563ef87e8878a926f238a46f74b5833f60b2415a2 WatchSource:0}: Error finding container eba4982a02d2008c9b6102e563ef87e8878a926f238a46f74b5833f60b2415a2: Status 404 returned error can't find the container with id eba4982a02d2008c9b6102e563ef87e8878a926f238a46f74b5833f60b2415a2 Nov 01 22:50:12.137040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.137018 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:50:12.140708 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.140663 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:12.140841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.140765 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:12.140841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.140827 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:12.468074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.463380 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f" path="/var/lib/kubelet/pods/9fd48ea2-0a2c-40ef-8d0f-99b8b5ed550f/volumes" Nov 01 22:50:12.566281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.566238 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" event={"ID":"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3","Type":"ContainerStarted","Data":"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f"} Nov 01 22:50:12.567257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.567229 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" event={"ID":"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3","Type":"ContainerStarted","Data":"eba4982a02d2008c9b6102e563ef87e8878a926f238a46f74b5833f60b2415a2"} Nov 01 22:50:12.622880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.620266 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podStartSLOduration=19.620241504 podStartE2EDuration="19.620241504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:12.602749005 +0000 UTC m=+1582.378735835" watchObservedRunningTime="2026-11-01 22:50:12.620241504 +0000 UTC m=+1582.396228334" Nov 01 22:50:12.737751 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:12.737660 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:13.745946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:13.745678 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:13.745946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:13.745745 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:14.738328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:14.738286 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:14.738654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:14.738623 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:15.742001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:15.738273 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:15.742001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:15.738337 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:16.744660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:16.738955 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:16.744660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:16.739024 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:17.475347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:17.473551 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:50:17.475347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:17.473995 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" containerID="cri-o://aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2" gracePeriod=30 Nov 01 22:50:17.776938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:17.742037 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:17.776938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:17.742114 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:18.738878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:18.738366 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:18.738878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:18.738435 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:19.003935 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:50:18.986054 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:50:19.739081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:19.739042 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:50:19.739310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:19.739114 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 22:50:20.745166 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:20.745131 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:20.746761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:20.746731 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:20.748717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:20.748692 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:50:22.921048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.921017 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:50:22.965468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.965420 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 22:50:22.966827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.966801 5778 cpu_manager.go:414] "containerMap: removing container" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" Nov 01 22:50:22.966963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.966830 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" Nov 01 22:50:22.967128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.967111 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerName="istio-proxy" Nov 01 22:50:22.968478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.968447 5778 generic.go:356] "Generic (PLEG): container finished" podID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" containerID="aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2" exitCode=0 Nov 01 22:50:22.968620 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.968561 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" Nov 01 22:50:22.971615 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.971587 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" event={"ID":"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc","Type":"ContainerDied","Data":"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2"} Nov 01 22:50:22.971755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.971627 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m" event={"ID":"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc","Type":"ContainerDied","Data":"9858eb3e714ffc1bbe05a2c807d4fd03d6369750f68f108dabb372b40e6a73b3"} Nov 01 22:50:22.971755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.971649 5778 scope.go:122] "RemoveContainer" containerID="aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2" Nov 01 22:50:22.971844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.971800 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:22.995440 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.990786 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 22:50:22.997078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.997059 5778 scope.go:122] "RemoveContainer" containerID="aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2" Nov 01 22:50:22.997431 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:50:22.997410 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2\": container with ID starting with aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2 not found: ID does not exist" containerID="aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2" Nov 01 22:50:22.997532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:22.997441 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2"} err="failed to get container status \"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2\": rpc error: code = NotFound desc = could not find container \"aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2\": container with ID starting with aa58ab6e9707ba95f1322d16f9be2f8ff1fdf85485af6ad486cd2a1b5e864ad2 not found: ID does not exist" Nov 01 22:50:23.010489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.010468 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c49240-openshift-default-dockercfg-chdtv\"" Nov 01 22:50:23.066821 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.066782 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067003 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067060 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067099 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067119 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.067253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067176 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067215 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067286 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067442 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.067486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067451 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067454 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.067486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067482 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\") pod \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\" (UID: \"26e67e68-5ca5-4994-b8c3-37c3e9ba42cc\") " Nov 01 22:50:23.067703 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067663 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.067766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067751 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.067827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067751 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:23.067943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067826 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.067943 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.067913 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.068203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.068178 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.068445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.068354 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.068617 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.068594 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.068757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.068702 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.068872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.068836 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.069050 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069029 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp44m\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.069275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069255 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.069378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069283 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.069378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069299 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.069378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069313 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.069378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069327 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.069746 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.069721 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "kube-api-access-5b5nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.070145 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.070120 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:23.070560 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.070542 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.070707 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.070691 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy" pod "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" (UID: "26e67e68-5ca5-4994-b8c3-37c3e9ba42cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.170290 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170254 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170416 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170508 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170483 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170602 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170652 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170734 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170735 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170823 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.170966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.170925 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp44m\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.171069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171032 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.171147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171078 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.171147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171092 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.171147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171110 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.171313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171234 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.171313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171251 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.171313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171269 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5nz\" (UniqueName: \"kubernetes.io/projected/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-kube-api-access-5b5nz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.171313 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171288 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.171479 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.171307 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.173481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.173457 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.173706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.173689 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.179249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.179227 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.179345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.179282 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp44m\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\") pod \"e2e-inf-a-c49240-openshift-default-67dbc89555-df94r\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.297205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.297168 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:50:23.301646 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.301619 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-98g2m"] Nov 01 22:50:23.304187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.304173 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:23.615265 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:50:23.615220 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cf342e_f2a6_4679_b075_ba699c16471e.slice/crio-95b330f16f9a2f35d77cb9df501af83f3cf903eec944c9d1f815c2b49f663532 WatchSource:0}: Error finding container 95b330f16f9a2f35d77cb9df501af83f3cf903eec944c9d1f815c2b49f663532: Status 404 returned error can't find the container with id 95b330f16f9a2f35d77cb9df501af83f3cf903eec944c9d1f815c2b49f663532 Nov 01 22:50:23.627120 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.627063 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:23.627394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.627370 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:23.627607 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.627587 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:50:23.633845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.633820 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 22:50:23.981335 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.981297 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerStarted","Data":"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967"} Nov 01 22:50:23.981887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:23.981365 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerStarted","Data":"95b330f16f9a2f35d77cb9df501af83f3cf903eec944c9d1f815c2b49f663532"} Nov 01 22:50:24.008398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.008337 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podStartSLOduration=6.008317603 podStartE2EDuration="6.008317603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:24.006145921 +0000 UTC m=+1593.782132759" watchObservedRunningTime="2026-11-01 22:50:24.008317603 +0000 UTC m=+1593.784304433" Nov 01 22:50:24.305329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.305244 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:24.321232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.321195 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:24.377918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.377822 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e67e68-5ca5-4994-b8c3-37c3e9ba42cc" path="/var/lib/kubelet/pods/26e67e68-5ca5-4994-b8c3-37c3e9ba42cc/volumes" Nov 01 22:50:24.991991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.991962 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:24.992920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:24.992836 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:50:41.022123 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.022087 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:50:41.027121 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.027097 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.032374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.032348 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b78bd\"/\"default-dockercfg-fpsrl\"" Nov 01 22:50:41.033689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.033667 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b78bd\"/\"test-model-6b78bd-kserve-self-signed-certs\"" Nov 01 22:50:41.038187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.036555 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:50:41.040300 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.040267 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b78bd\"/\"kube-root-ca.crt\"" Nov 01 22:50:41.040469 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.040453 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b78bd\"/\"openshift-service-ca.crt\"" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.080886 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.080950 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.080987 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmn8\" (UniqueName: \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.081054 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.081129 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.082956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.081346 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.182963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.182926 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183119 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183168 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183306 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183201 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmn8\" (UniqueName: \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183306 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183272 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183350 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183466 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183418 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183498 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.183731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.183704 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.186593 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.186571 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.186959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.186931 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.193455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.193427 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmn8\" (UniqueName: \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\") pod \"test-model-6b78bd-kserve-5cbf665d9-mj4qh\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:41.350393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:41.350294 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:42.118523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:42.118489 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:50:42.122106 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:50:42.122061 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9caa97f7_2b0a_44b0_bc8d_3b32c82b5a35.slice/crio-a3f61b702aa2d0abb0e39a570aa8a1db0001d2c4cbf152d0a4bab92ae1a77c62 WatchSource:0}: Error finding container a3f61b702aa2d0abb0e39a570aa8a1db0001d2c4cbf152d0a4bab92ae1a77c62: Status 404 returned error can't find the container with id a3f61b702aa2d0abb0e39a570aa8a1db0001d2c4cbf152d0a4bab92ae1a77c62 Nov 01 22:50:42.170502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:42.169943 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" event={"ID":"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35","Type":"ContainerStarted","Data":"a3f61b702aa2d0abb0e39a570aa8a1db0001d2c4cbf152d0a4bab92ae1a77c62"} Nov 01 22:50:43.179819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:43.179773 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" event={"ID":"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35","Type":"ContainerStarted","Data":"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b"} Nov 01 22:50:43.202653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:43.202600 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podStartSLOduration=3.2025844230000002 podStartE2EDuration="3.202584423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:43.199841141 +0000 UTC m=+1612.975827972" watchObservedRunningTime="2026-11-01 22:50:43.202584423 +0000 UTC m=+1612.978571251" Nov 01 22:50:43.401588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:43.401201 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:48.777642 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:50:48.777599 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:50:51.352878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:51.352283 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:51.448876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:51.447902 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:52.322326 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:52.322291 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:50:57.576640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.576603 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:50:57.587744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.587714 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.599330 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.599296 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e685d\"/\"default-dockercfg-26q64\"" Nov 01 22:50:57.600549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.600524 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e685d\"/\"kube-root-ca.crt\"" Nov 01 22:50:57.600963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.600943 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e685d\"/\"openshift-service-ca.crt\"" Nov 01 22:50:57.601278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.601259 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e685d\"/\"test-model-0e685d-kserve-self-signed-certs\"" Nov 01 22:50:57.606240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.605584 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:50:57.709800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.709739 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.710029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.709980 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.710182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.710162 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.710251 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.710223 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.710311 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.710283 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fm8\" (UniqueName: \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.710392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.710375 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.818919 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.819007 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.819050 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fm8\" (UniqueName: \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.819168 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.819277 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.819439 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.820889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.820454 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.823887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.821423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.856811 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.840346 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.889877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.871539 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.889877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.872149 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fm8\" (UniqueName: \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.889877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.873116 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\") pod \"test-model-0e685d-kserve-c6cbcf699-7gf5j\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:57.918583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:57.917231 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:50:58.406338 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:58.405076 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:50:58.408921 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:50:58.408817 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55a779fb_eaca_4e55_b27d_44d504bce1d3.slice/crio-d87c4b9019d78f6f0ee9b72c5d589c451d91be390c4bc3545d5474c93d83281b WatchSource:0}: Error finding container d87c4b9019d78f6f0ee9b72c5d589c451d91be390c4bc3545d5474c93d83281b: Status 404 returned error can't find the container with id d87c4b9019d78f6f0ee9b72c5d589c451d91be390c4bc3545d5474c93d83281b Nov 01 22:50:59.435190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:59.435151 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" event={"ID":"55a779fb-eaca-4e55-b27d-44d504bce1d3","Type":"ContainerStarted","Data":"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de"} Nov 01 22:50:59.437565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:59.437533 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" event={"ID":"55a779fb-eaca-4e55-b27d-44d504bce1d3","Type":"ContainerStarted","Data":"d87c4b9019d78f6f0ee9b72c5d589c451d91be390c4bc3545d5474c93d83281b"} Nov 01 22:50:59.460515 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:59.460456 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podStartSLOduration=2.460437936 podStartE2EDuration="2.460437936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:59.458587601 +0000 UTC m=+1629.234574431" watchObservedRunningTime="2026-11-01 22:50:59.460437936 +0000 UTC m=+1629.236424767" Nov 01 22:50:59.917965 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:50:59.917925 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:07.917737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:07.917690 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:07.937302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:07.937268 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:08.533437 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:08.533403 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:09.624061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:09.624019 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:51:09.624592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:09.624334 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" containerID="cri-o://21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68" gracePeriod=30 Nov 01 22:51:15.020478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.020447 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:51:15.061370 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.061323 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 22:51:15.064187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.062839 5778 cpu_manager.go:414] "containerMap: removing container" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" Nov 01 22:51:15.064187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.062884 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" Nov 01 22:51:15.064187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.063088 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" containerName="istio-proxy" Nov 01 22:51:15.069291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.069123 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.072516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.072487 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 22:51:15.108097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.108069 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39e952\"" Nov 01 22:51:15.190565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190531 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.190763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190588 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.190763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190640 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.190763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190713 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190886 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:15.191081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.190979 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:15.191184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191144 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191245 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191191 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191245 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191236 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191290 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191354 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\") pod \"52ec7b7a-a00c-4206-98b1-034896709f93\" (UID: \"52ec7b7a-a00c-4206-98b1-034896709f93\") " Nov 01 22:51:15.191913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191642 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:15.191913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191670 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.191913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191786 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:15.191913 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191813 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:15.192220 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.191968 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqpr\" (UniqueName: \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.192492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.192472 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ec7b7a-a00c-4206-98b1-034896709f93-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.192541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.192501 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.192541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.192516 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.192541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.192530 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.192652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.192543 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.193351 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.193325 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:15.193481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.193461 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:15.194238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.194221 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:51:15.194303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.194252 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j" pod "52ec7b7a-a00c-4206-98b1-034896709f93" (UID: "52ec7b7a-a00c-4206-98b1-034896709f93"). InnerVolumeSpecName "kube-api-access-26c7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:15.294253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294154 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqpr\" (UniqueName: \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.294591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294561 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.294735 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294721 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ec7b7a-a00c-4206-98b1-034896709f93-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.294787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294738 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.294787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294749 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26c7j\" (UniqueName: \"kubernetes.io/projected/52ec7b7a-a00c-4206-98b1-034896709f93-kube-api-access-26c7j\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.294787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.294758 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ec7b7a-a00c-4206-98b1-034896709f93-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:15.295250 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.295230 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.303501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.303478 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqpr\" (UniqueName: \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\") pod \"payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.405942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.405838 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:15.589602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.589498 5778 generic.go:356] "Generic (PLEG): container finished" podID="52ec7b7a-a00c-4206-98b1-034896709f93" containerID="21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68" exitCode=0 Nov 01 22:51:15.589602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.589543 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" Nov 01 22:51:15.589602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.589575 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" event={"ID":"52ec7b7a-a00c-4206-98b1-034896709f93","Type":"ContainerDied","Data":"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68"} Nov 01 22:51:15.590000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.589686 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz" event={"ID":"52ec7b7a-a00c-4206-98b1-034896709f93","Type":"ContainerDied","Data":"d0c8d72113209c3ea0abb4d22ce2ceb9c738c223eec788521e904462390daceb"} Nov 01 22:51:15.590000 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.589718 5778 scope.go:122] "RemoveContainer" containerID="21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68" Nov 01 22:51:15.604688 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.604665 5778 scope.go:122] "RemoveContainer" containerID="21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68" Nov 01 22:51:15.605081 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:51:15.605053 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68\": container with ID starting with 21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68 not found: ID does not exist" containerID="21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68" Nov 01 22:51:15.605196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.605092 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68"} err="failed to get container status \"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68\": rpc error: code = NotFound desc = could not find container \"21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68\": container with ID starting with 21f015907c3fa3f40abc4e61fe7e2268503da5751c4051ea7a2aa9413db61c68 not found: ID does not exist" Nov 01 22:51:15.686979 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.686937 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:51:15.703224 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.703169 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-7nztz"] Nov 01 22:51:15.707284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:15.707243 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 22:51:15.721050 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:51:15.721016 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf342bcc0_a2df_4dbe_9ec2_18b9e144d316.slice/crio-44babd1d5436552175354425fe82f58d1a06b7aeebf12cbd4c8c4ed3b03d257a WatchSource:0}: Error finding container 44babd1d5436552175354425fe82f58d1a06b7aeebf12cbd4c8c4ed3b03d257a: Status 404 returned error can't find the container with id 44babd1d5436552175354425fe82f58d1a06b7aeebf12cbd4c8c4ed3b03d257a Nov 01 22:51:16.377173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:16.377138 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ec7b7a-a00c-4206-98b1-034896709f93" path="/var/lib/kubelet/pods/52ec7b7a-a00c-4206-98b1-034896709f93/volumes" Nov 01 22:51:16.601458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:16.601420 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" event={"ID":"f342bcc0-a2df-4dbe-9ec2-18b9e144d316","Type":"ContainerStarted","Data":"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e"} Nov 01 22:51:16.601673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:16.601486 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:16.601673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:16.601509 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" event={"ID":"f342bcc0-a2df-4dbe-9ec2-18b9e144d316","Type":"ContainerStarted","Data":"44babd1d5436552175354425fe82f58d1a06b7aeebf12cbd4c8c4ed3b03d257a"} Nov 01 22:51:16.618570 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:16.618522 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" podStartSLOduration=47.618506981 podStartE2EDuration="47.618506981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:16.617257401 +0000 UTC m=+1646.393244230" watchObservedRunningTime="2026-11-01 22:51:16.618506981 +0000 UTC m=+1646.394493811" Nov 01 22:51:18.777241 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:51:18.777197 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:51:27.614467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:27.614436 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 22:51:35.224669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.222636 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.294921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.290968 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.320935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.315967 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.538039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.537937 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.574151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.574092 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.692723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.692001 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.761092 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.761006 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:35.906181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:35.906116 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.015877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.001203 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded" Nov 01 22:51:36.028994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.028939 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.166940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.165972 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.170876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.167960 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:51:36.207887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.206954 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.249872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.230288 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.249872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.231085 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:51:36.294934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.292958 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.294934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.293525 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:51:36.337929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.317934 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.337929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.321066 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:51:36.541993 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.538762 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.653330 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.585955 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.657910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.656444 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.699959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.692149 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.129:8000/ready\": context deadline exceeded" Nov 01 22:51:36.998494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:36.998445 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded" Nov 01 22:51:37.112015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.111969 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.201744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.201699 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271356 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271433 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271464 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271493 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271523 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271553 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271582 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:51:37.313977 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.271611 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 22:51:37.328244 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.319944 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.412969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.412937 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:51:37.414234 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.414204 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:51:37.480928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:37.480833 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:51:47.595076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.595037 5778 generic.go:356] "Generic (PLEG): container finished" podID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerID="063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967" exitCode=0 Nov 01 22:51:47.595654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.595082 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerDied","Data":"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967"} Nov 01 22:51:47.595654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.595616 5778 scope.go:122] "RemoveContainer" containerID="063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967" Nov 01 22:51:47.597894 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.597829 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:51:47.597994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.597943 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:51:47.598040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:47.597998 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:51:48.606094 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:48.606052 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerStarted","Data":"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703"} Nov 01 22:51:48.801593 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:51:48.801561 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:51:49.309880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:49.306287 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:51:49.309880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:49.306435 5778 patch_prober.go:29] interesting pod/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:51:49.309880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:49.306480 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 22:51:50.305292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:50.305261 5778 patch_prober.go:29] interesting pod/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:51:50.305748 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:50.305324 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 22:51:51.311874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:51.306410 5778 patch_prober.go:29] interesting pod/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:51:51.311874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:51.306486 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 22:51:52.305016 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:52.304978 5778 patch_prober.go:29] interesting pod/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:51:52.305218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:52.305038 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 22:51:53.304602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:53.304564 5778 patch_prober.go:29] interesting pod/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:51:53.305187 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:53.304619 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 22:51:54.312012 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:54.311982 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:51:54.313204 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:54.313175 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:51:54.314366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:54.314193 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 22:51:55.605408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:55.605374 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:51:55.605831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:51:55.605588 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" containerID="cri-o://3f16f8da0dca129afcd8c1af52cc2b116c939128c26d560e5bc3af1d7d921eb6" gracePeriod=30 Nov 01 22:52:00.798128 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:00.798085 5778 generic.go:356] "Generic (PLEG): container finished" podID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerID="3f16f8da0dca129afcd8c1af52cc2b116c939128c26d560e5bc3af1d7d921eb6" exitCode=0 Nov 01 22:52:00.798572 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:00.798159 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" event={"ID":"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f","Type":"ContainerDied","Data":"3f16f8da0dca129afcd8c1af52cc2b116c939128c26d560e5bc3af1d7d921eb6"} Nov 01 22:52:01.021220 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.021187 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:52:01.070320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.070277 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:52:01.075257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.075216 5778 cpu_manager.go:414] "containerMap: removing container" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" Nov 01 22:52:01.075257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.075256 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" Nov 01 22:52:01.075921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.075895 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" containerName="istio-proxy" Nov 01 22:52:01.088168 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.087996 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.095918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.095889 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1799f8-openshift-default-dockercfg-fvhdd\"" Nov 01 22:52:01.096635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.096599 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.197525 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.197620 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.198407 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.198489 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.198590 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.198697 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.198790 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199006 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199082 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\") pod \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\" (UID: \"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f\") " Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199472 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199558 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199590 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199695 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199724 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199764 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199890 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nc8\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.200003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.199977 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.201233 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.200111 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.202222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.202169 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.203806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.203322 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.204272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.204248 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:01.204458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.204432 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.204551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.204502 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.205248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.205225 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "kube-api-access-zsrrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:01.206629 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.206599 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:01.207118 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.207095 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.207212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.207152 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token" pod "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" (UID: "15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:01.302090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302053 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302096 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302180 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302213 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302244 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302324 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nc8\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302390 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302494 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302502 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302545 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302567 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302749 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.302956 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302797 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302963 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302981 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.302997 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303012 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303025 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303039 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303051 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303066 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303081 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrrw\" (UniqueName: \"kubernetes.io/projected/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f-kube-api-access-zsrrw\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.303256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.303047 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.305201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.305177 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.305487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.305468 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.312013 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.311986 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.312262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.312244 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nc8\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.429495 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.429462 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:01.747710 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.747314 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:52:01.774946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.765019 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:52:01.775339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.775304 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:52:01.776087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.776042 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:52:01.810780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.810744 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" event={"ID":"15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f","Type":"ContainerDied","Data":"0c457ed692ab93e8fc31a19cb3d6350b72f9cad6dae1ea6672d916853da4ac89"} Nov 01 22:52:01.811447 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.810802 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk" Nov 01 22:52:01.811447 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.810826 5778 scope.go:122] "RemoveContainer" containerID="3f16f8da0dca129afcd8c1af52cc2b116c939128c26d560e5bc3af1d7d921eb6" Nov 01 22:52:01.813779 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.813747 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" event={"ID":"eb98f5d2-0a05-479a-887a-f504e20bcd89","Type":"ContainerStarted","Data":"c2d7acafcbb5a902209e568d6d8064d49dffee74d6e020ac0741108460abdf2c"} Nov 01 22:52:01.857598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.857298 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:52:01.864985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:01.864950 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-2g5xk"] Nov 01 22:52:02.378573 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:02.378539 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f" path="/var/lib/kubelet/pods/15d76b9e-cb48-4bf3-aab0-ed31dfd3ef6f/volumes" Nov 01 22:52:02.861774 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:02.861734 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" event={"ID":"eb98f5d2-0a05-479a-887a-f504e20bcd89","Type":"ContainerStarted","Data":"3ebe687db2b09e0c25bcfba6f9441a9222408e47b6a4a3e0217d79c3dea68a74"} Nov 01 22:52:02.887111 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:02.887052 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podStartSLOduration=2.887031994 podStartE2EDuration="2.887031994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:02.883189217 +0000 UTC m=+1692.659176047" watchObservedRunningTime="2026-11-01 22:52:02.887031994 +0000 UTC m=+1692.663018824" Nov 01 22:52:03.433876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:03.432029 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:03.454590 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:03.454056 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:03.454590 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:03.454153 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:04.430878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:04.430348 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:04.430878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:04.430413 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:05.432887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:05.432243 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:05.432887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:05.432314 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:06.432879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:06.432281 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:06.432879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:06.432347 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:07.446541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:07.446495 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:07.447149 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:07.446573 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:08.430827 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:08.430784 5778 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:52:08.431213 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:08.431171 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 22:52:09.487270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:09.487238 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:09.488449 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:09.488421 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:09.490151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:09.490124 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:52:10.384411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:10.384377 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:52:10.384618 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:10.384593 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" containerID="cri-o://db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f" gracePeriod=30 Nov 01 22:52:10.395921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:10.395890 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:52:10.396142 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:10.396118 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" containerID="cri-o://874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32" gracePeriod=30 Nov 01 22:52:12.411117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:12.411078 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:52:12.411528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:12.411147 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:52:15.796728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.796697 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.847008 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.847102 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.847173 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.847637 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848264 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848350 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848425 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848470 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848519 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.848551 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\") pod \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\" (UID: \"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd\") " Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.850267 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.850502 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.850985 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.851230 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.852901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.851399 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.868025 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.868085 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.869395 5778 cpu_manager.go:414] "containerMap: removing container" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.869415 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.869666 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerName="istio-proxy" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.870051 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.877656 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.878094 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.878590 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.880275 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.878735 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc" pod "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" (UID: "6c7b01d1-72f2-44e6-8ea2-dc292e43fffd"). InnerVolumeSpecName "kube-api-access-hn4mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.882127 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.882077 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b48ab8-serving-cert\"" Nov 01 22:52:15.888059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.888031 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:52:15.938876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.933150 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 22:52:15.938876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.934679 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" Nov 01 22:52:15.938876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.934701 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" Nov 01 22:52:15.938876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.935019 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerName="istio-proxy" Nov 01 22:52:15.953967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.941402 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:15.953967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.946340 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 22:52:15.954324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954301 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954401 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954491 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954479 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954517 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954545 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954653 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954585 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.954839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954824 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.955003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.954988 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.955069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955038 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\") pod \"5254fd0a-d17d-448d-8195-7adfa483b6a7\" (UID: \"5254fd0a-d17d-448d-8195-7adfa483b6a7\") " Nov 01 22:52:15.955317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955298 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:15.955427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955410 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:15.955489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955447 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:15.955562 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955549 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jp4\" (UniqueName: \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:15.955687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955675 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:15.955749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955733 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:15.955842 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.955830 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:15.956151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956130 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cnl\" (UniqueName: \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:15.956439 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956424 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956444 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956459 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4mc\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-kube-api-access-hn4mc\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956476 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956490 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956503 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956518 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.956793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.956531 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.961350 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.957651 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:15.961350 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.958008 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.964992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.964968 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:15.966032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.966008 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.966205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.966183 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.966340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.966321 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.966460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.966446 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.973925 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.973901 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.979011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.978987 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5" pod "5254fd0a-d17d-448d-8195-7adfa483b6a7" (UID: "5254fd0a-d17d-448d-8195-7adfa483b6a7"). InnerVolumeSpecName "kube-api-access-vrpj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.982404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:15.982387 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39e952-serving-cert\"" Nov 01 22:52:16.038494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.038455 5778 generic.go:356] "Generic (PLEG): container finished" podID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" containerID="db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f" exitCode=0 Nov 01 22:52:16.038716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.038696 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" event={"ID":"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd","Type":"ContainerDied","Data":"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f"} Nov 01 22:52:16.038781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.038755 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" event={"ID":"6c7b01d1-72f2-44e6-8ea2-dc292e43fffd","Type":"ContainerDied","Data":"853726c2affe46d0773c48e60e1ff6026d21b65cdd3a376c401d4424360b076c"} Nov 01 22:52:16.038819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.038783 5778 scope.go:122] "RemoveContainer" containerID="db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f" Nov 01 22:52:16.038931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.038918 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg" Nov 01 22:52:16.042647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.042618 5778 generic.go:356] "Generic (PLEG): container finished" podID="5254fd0a-d17d-448d-8195-7adfa483b6a7" containerID="874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32" exitCode=0 Nov 01 22:52:16.042783 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.042748 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" event={"ID":"5254fd0a-d17d-448d-8195-7adfa483b6a7","Type":"ContainerDied","Data":"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32"} Nov 01 22:52:16.042962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.042830 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" event={"ID":"5254fd0a-d17d-448d-8195-7adfa483b6a7","Type":"ContainerDied","Data":"0cfe756226c72bb585ed2b25db4b05f6eef87d82c8a3c9fe55fe75dda769f473"} Nov 01 22:52:16.042962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.042911 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj" Nov 01 22:52:16.058912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.058885 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.059011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.058933 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.059122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059010 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jp4\" (UniqueName: \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.059122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059111 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.059264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059155 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.059264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059228 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.059382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059281 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cnl\" (UniqueName: \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.059444 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059395 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.059492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059482 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5254fd0a-d17d-448d-8195-7adfa483b6a7-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059497 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059509 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059520 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059531 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059545 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpj5\" (UniqueName: \"kubernetes.io/projected/5254fd0a-d17d-448d-8195-7adfa483b6a7-kube-api-access-vrpj5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059561 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059574 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.059712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.059586 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5254fd0a-d17d-448d-8195-7adfa483b6a7-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:16.060308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.060281 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.062487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.062464 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.063074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.063041 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.063299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.063203 5778 scope.go:122] "RemoveContainer" containerID="db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f" Nov 01 22:52:16.063939 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:52:16.063525 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f\": container with ID starting with db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f not found: ID does not exist" containerID="db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f" Nov 01 22:52:16.063939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.063558 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f"} err="failed to get container status \"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f\": rpc error: code = NotFound desc = could not find container \"db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f\": container with ID starting with db714c36981716f2373a0fe4ce4c98c97be1eb68c8ae6ce2180f980d56dca28f not found: ID does not exist" Nov 01 22:52:16.063939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.063581 5778 scope.go:122] "RemoveContainer" containerID="874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32" Nov 01 22:52:16.064395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.064373 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.064492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.064476 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.066980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.066100 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.072506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.072429 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cnl\" (UniqueName: \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\") pod \"maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.079400 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.079378 5778 scope.go:122] "RemoveContainer" containerID="874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32" Nov 01 22:52:16.079799 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:52:16.079771 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32\": container with ID starting with 874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32 not found: ID does not exist" containerID="874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32" Nov 01 22:52:16.079950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.079811 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32"} err="failed to get container status \"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32\": rpc error: code = NotFound desc = could not find container \"874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32\": container with ID starting with 874944f7f2792422f4bededa102d0d3969a0d129ed49dac99a6f5da0400dab32 not found: ID does not exist" Nov 01 22:52:16.079950 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.079787 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jp4\" (UniqueName: \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\") pod \"maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.082588 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.082568 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:52:16.088839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.088816 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-2w4qg"] Nov 01 22:52:16.108176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.108147 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:52:16.112582 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.112554 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-x2dxj"] Nov 01 22:52:16.207160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.207085 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:16.283366 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.283030 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:16.390696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.390593 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5254fd0a-d17d-448d-8195-7adfa483b6a7" path="/var/lib/kubelet/pods/5254fd0a-d17d-448d-8195-7adfa483b6a7/volumes" Nov 01 22:52:16.392409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.391566 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7b01d1-72f2-44e6-8ea2-dc292e43fffd" path="/var/lib/kubelet/pods/6c7b01d1-72f2-44e6-8ea2-dc292e43fffd/volumes" Nov 01 22:52:16.609457 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:52:16.609421 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35967fa_2928_4dee_a7f5_ee70e8a36b5e.slice/crio-1ddb82333e0da6727b66ff85e479167cce7cd525e5d7b93e78f0b1e0ad616793 WatchSource:0}: Error finding container 1ddb82333e0da6727b66ff85e479167cce7cd525e5d7b93e78f0b1e0ad616793: Status 404 returned error can't find the container with id 1ddb82333e0da6727b66ff85e479167cce7cd525e5d7b93e78f0b1e0ad616793 Nov 01 22:52:16.610422 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.610382 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 22:52:16.611574 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.611552 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:52:16.645932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:16.645894 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 22:52:16.646088 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:52:16.646028 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993c305_4768_45c5_a8db_2e7e14d13c8f.slice/crio-f1b62d4d7cf5d95bc106bfb8626124205b88261ddaca9593051333a1d9d4aeed WatchSource:0}: Error finding container f1b62d4d7cf5d95bc106bfb8626124205b88261ddaca9593051333a1d9d4aeed: Status 404 returned error can't find the container with id f1b62d4d7cf5d95bc106bfb8626124205b88261ddaca9593051333a1d9d4aeed Nov 01 22:52:17.074454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:17.074393 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" event={"ID":"b35967fa-2928-4dee-a7f5-ee70e8a36b5e","Type":"ContainerStarted","Data":"1ddb82333e0da6727b66ff85e479167cce7cd525e5d7b93e78f0b1e0ad616793"} Nov 01 22:52:17.077192 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:17.077158 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" event={"ID":"f993c305-4768-45c5-a8db-2e7e14d13c8f","Type":"ContainerStarted","Data":"f1b62d4d7cf5d95bc106bfb8626124205b88261ddaca9593051333a1d9d4aeed"} Nov 01 22:52:18.088866 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.088814 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" event={"ID":"b35967fa-2928-4dee-a7f5-ee70e8a36b5e","Type":"ContainerStarted","Data":"2cafecf1ef93b08f9812ee25dda2299044d65ba2a296ebd657d7b41d1c270279"} Nov 01 22:52:18.089359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.088985 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:18.090593 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.090566 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" event={"ID":"f993c305-4768-45c5-a8db-2e7e14d13c8f","Type":"ContainerStarted","Data":"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d"} Nov 01 22:52:18.090712 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.090695 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:18.118053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.117991 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" podStartSLOduration=86.656953328 podStartE2EDuration="1m27.117971018s" totalImagesPullingTime="461.01769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:18.114365971 +0000 UTC m=+1707.890352825" watchObservedRunningTime="2026-11-01 22:52:18.117971018 +0000 UTC m=+1707.893957849" Nov 01 22:52:18.140492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:18.140439 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" podStartSLOduration=85.623013765 podStartE2EDuration="1m26.140422452s" totalImagesPullingTime="517.408687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:18.13579639 +0000 UTC m=+1707.911783220" watchObservedRunningTime="2026-11-01 22:52:18.140422452 +0000 UTC m=+1707.916409280" Nov 01 22:52:18.776778 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:52:18.776743 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:52:24.104523 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:24.104492 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 22:52:24.104945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:24.104901 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 22:52:30.472530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.472440 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 22:52:30.485924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.479685 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.545294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.545058 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 22:52:30.581519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.580913 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-021714\"/\"default-dockercfg-stjc8\"" Nov 01 22:52:30.581519 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.581123 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-021714\"/\"test-model-021714-kserve-self-signed-certs\"" Nov 01 22:52:30.583360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.583335 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-021714\"/\"kube-root-ca.crt\"" Nov 01 22:52:30.583536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.583517 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-021714\"/\"openshift-service-ca.crt\"" Nov 01 22:52:30.671766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.671715 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.671766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.671763 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.672108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.671991 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.672108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.672015 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.672108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.672049 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.672281 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.672225 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2sbh\" (UniqueName: \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775018 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775075 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775114 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775181 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2sbh\" (UniqueName: \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775412 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.775446 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.776504 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.776749 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.778928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.777577 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.790465 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.790428 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.797874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.794144 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2sbh\" (UniqueName: \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.798696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.798669 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\") pod \"test-model-021714-kserve-8d45fb497-qbn5g\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:30.853230 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:30.853193 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:31.302345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:31.302312 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 22:52:31.307528 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:52:31.307496 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0989540_2365_40ec_a84c_1e1263072037.slice/crio-77513b6314413fba98431ae381ef68be1d91fb3a289f496ba3d5b70b6dd5aa72 WatchSource:0}: Error finding container 77513b6314413fba98431ae381ef68be1d91fb3a289f496ba3d5b70b6dd5aa72: Status 404 returned error can't find the container with id 77513b6314413fba98431ae381ef68be1d91fb3a289f496ba3d5b70b6dd5aa72 Nov 01 22:52:32.273929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:32.273887 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" event={"ID":"d0989540-2365-40ec-a84c-1e1263072037","Type":"ContainerStarted","Data":"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0"} Nov 01 22:52:32.274362 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:32.273940 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" event={"ID":"d0989540-2365-40ec-a84c-1e1263072037","Type":"ContainerStarted","Data":"77513b6314413fba98431ae381ef68be1d91fb3a289f496ba3d5b70b6dd5aa72"} Nov 01 22:52:32.297315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:32.297248 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" podStartSLOduration=2.297227373 podStartE2EDuration="2.297227373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:32.29211875 +0000 UTC m=+1722.068105580" watchObservedRunningTime="2026-11-01 22:52:32.297227373 +0000 UTC m=+1722.073214205" Nov 01 22:52:32.854634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:32.854601 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:40.854786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:40.854746 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:40.867964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:40.867896 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:41.479303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:41.479256 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 22:52:43.983623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:43.983576 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 22:52:43.991654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:43.990186 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.006907 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.004722 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 22:52:44.009279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.007263 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39e952\"/\"kube-root-ca.crt\"" Nov 01 22:52:44.009279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.008556 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39e952\"/\"openshift-service-ca.crt\"" Nov 01 22:52:44.009279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.008656 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39e952\"/\"test-model-39e952-kserve-self-signed-certs\"" Nov 01 22:52:44.009279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.008742 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39e952\"/\"default-dockercfg-q5gtg\"" Nov 01 22:52:44.087141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087089 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngvq\" (UniqueName: \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.087324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087200 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.087324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087287 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.087411 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087344 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.087628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087602 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.087733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.087714 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196271 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196506 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196563 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196721 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngvq\" (UniqueName: \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196764 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.196887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.196882 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.211872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.206338 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.211872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.206619 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.211872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.206965 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.230009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.229974 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.262014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.240638 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.262014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.254618 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngvq\" (UniqueName: \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\") pod \"test-model-39e952-kserve-6b757bd885-jndwf\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.317875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.317167 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:44.840598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:44.840572 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 22:52:45.544878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:45.544291 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" event={"ID":"d9ccfea0-0f8e-460e-9328-15462d663175","Type":"ContainerStarted","Data":"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f"} Nov 01 22:52:45.544878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:45.544347 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" event={"ID":"d9ccfea0-0f8e-460e-9328-15462d663175","Type":"ContainerStarted","Data":"faa76da975b5848250a81b44b0c620c3e8b7d5d27f0f6f2ac38bf2f55207ee62"} Nov 01 22:52:45.590211 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:45.588261 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" podStartSLOduration=2.588244237 podStartE2EDuration="2.588244237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:45.577004871 +0000 UTC m=+1735.352991700" watchObservedRunningTime="2026-11-01 22:52:45.588244237 +0000 UTC m=+1735.364231065" Nov 01 22:52:46.323219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:46.323118 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:48.777517 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:52:48.777471 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:52:53.216539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.216489 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 22:52:53.243905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.243813 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 22:52:53.244163 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.243940 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.250183 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.250156 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f675d\"/\"openshift-service-ca.crt\"" Nov 01 22:52:53.250445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.250429 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f675d\"/\"kube-root-ca.crt\"" Nov 01 22:52:53.250731 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.250713 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f675d\"/\"test-model-2f675d-kserve-self-signed-certs\"" Nov 01 22:52:53.251040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.251023 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f675d\"/\"default-dockercfg-fggp5\"" Nov 01 22:52:53.402966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.402930 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.403167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.403144 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.403501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.403480 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.403644 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.403629 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.403967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.403938 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxcn\" (UniqueName: \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.404299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.404277 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.520935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.507055 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.527159 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.533121 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxcn\" (UniqueName: \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.533337 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.533386 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.533431 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.511504 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.544875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.535274 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.585413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.585341 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.608881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.595706 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxcn\" (UniqueName: \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.608881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.607228 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.673176 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.644588 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\") pod \"test-model-2f675d-kserve-67d7bc44f9-tmg9n\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:53.889986 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:53.889892 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:54.279797 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:52:54.279761 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf89e722b_3638_42dc_afdc_76afef5f3159.slice/crio-753bac8baee0afe301896c78a152a6b11511766d0ec5bfc945c4e8d386a79db8 WatchSource:0}: Error finding container 753bac8baee0afe301896c78a152a6b11511766d0ec5bfc945c4e8d386a79db8: Status 404 returned error can't find the container with id 753bac8baee0afe301896c78a152a6b11511766d0ec5bfc945c4e8d386a79db8 Nov 01 22:52:54.285664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.285636 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 22:52:54.318900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.318254 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:54.339389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.339358 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:54.786885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.786805 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" event={"ID":"f89e722b-3638-42dc-afdc-76afef5f3159","Type":"ContainerStarted","Data":"be85e823211c4db8c909ffad9448701b1ac1dd6fc20e63ea67f9a85e99b1f5a8"} Nov 01 22:52:54.786885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.786872 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" event={"ID":"f89e722b-3638-42dc-afdc-76afef5f3159","Type":"ContainerStarted","Data":"753bac8baee0afe301896c78a152a6b11511766d0ec5bfc945c4e8d386a79db8"} Nov 01 22:52:54.798725 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.798691 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:52:54.806521 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.806465 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" podStartSLOduration=1.8064438310000002 podStartE2EDuration="1.806443831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:54.804300459 +0000 UTC m=+1744.580287289" watchObservedRunningTime="2026-11-01 22:52:54.806443831 +0000 UTC m=+1744.582430662" Nov 01 22:52:54.897555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:54.897526 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:52:55.485076 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:55.485035 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:52:55.485605 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:52:55.485305 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" containerID="cri-o://c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f" gracePeriod=30 Nov 01 22:53:00.383936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.383905 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:53:00.384355 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.383957 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:53:00.924967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.924520 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:53:00.925754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.925726 5778 generic.go:356] "Generic (PLEG): container finished" podID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerID="c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f" exitCode=0 Nov 01 22:53:00.926309 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.926281 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" event={"ID":"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd","Type":"ContainerDied","Data":"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f"} Nov 01 22:53:00.926485 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.926467 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" event={"ID":"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd","Type":"ContainerDied","Data":"0c1c11ce44eb01c5a1aa7db7578a16d64a5b8dc0f2cbf915b94b7ebd65c0af87"} Nov 01 22:53:00.926607 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.926595 5778 scope.go:122] "RemoveContainer" containerID="c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f" Nov 01 22:53:00.942270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.942244 5778 scope.go:122] "RemoveContainer" containerID="c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f" Nov 01 22:53:00.942633 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:53:00.942599 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f\": container with ID starting with c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f not found: ID does not exist" containerID="c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f" Nov 01 22:53:00.942756 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.942635 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f"} err="failed to get container status \"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f\": rpc error: code = NotFound desc = could not find container \"c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f\": container with ID starting with c6bd086c256f92283bd0724712bf59e311d0b69b29547afe7800d890b0b2955f not found: ID does not exist" Nov 01 22:53:00.970571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.970538 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:53:00.971586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.971551 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" Nov 01 22:53:00.971586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.971579 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" Nov 01 22:53:00.971899 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.971877 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" containerName="istio-proxy" Nov 01 22:53:00.982215 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.982185 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:00.984636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:00.984609 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:53:01.049558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.049523 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.049772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.049569 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.049772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.049624 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.049772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.049685 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.049772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.049723 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.050055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050030 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.050116 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050054 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.050268 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050242 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.050424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050404 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:01.050643 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050624 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.050815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050800 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\") pod \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\" (UID: \"a3973874-a07c-43dc-ab4f-f81b3ed0c5dd\") " Nov 01 22:53:01.051180 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050679 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.051264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.050898 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.051264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.051088 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.053979 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.053952 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.054109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.053983 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.054109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.053999 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.054109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.054013 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.054109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.054027 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.054329 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.054199 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:01.054724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.054704 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "kube-api-access-kwznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:01.056071 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.056037 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:53:01.056574 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.056549 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy" pod "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" (UID: "a3973874-a07c-43dc-ab4f-f81b3ed0c5dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.155689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.155644 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.155689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.155692 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.155872 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.155926 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156138 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156090 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156193 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156228 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156411 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156468 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmwt\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.156698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156684 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.156765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156704 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.156765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156721 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwznl\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-kube-api-access-kwznl\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.156765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.156734 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.258245 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258146 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258270 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258299 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258426 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258599 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258481 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmwt\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258620 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258638 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258650 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258764 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.258804 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.258797 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.259184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.259140 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.259420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.259297 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.259420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.259333 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.259420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.259391 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.261308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.261286 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.261640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.261619 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.266606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.266581 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.266769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.266751 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmwt\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.315381 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.315349 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:01.681922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.681878 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:53:01.695212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.695094 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:01.695212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.695189 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:01.695541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.695232 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:01.943522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.943427 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" event={"ID":"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78","Type":"ContainerStarted","Data":"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa"} Nov 01 22:53:01.943522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.943498 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" event={"ID":"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78","Type":"ContainerStarted","Data":"bbb35b95ce5fef20c25129de0c6615228c6fd01cb4f7656bf6c83f6f4a65d8df"} Nov 01 22:53:01.946600 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.946567 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5" Nov 01 22:53:01.968008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.967955 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podStartSLOduration=128.967940675 podStartE2EDuration="2m8.967940675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:01.966229139 +0000 UTC m=+1751.742215958" watchObservedRunningTime="2026-11-01 22:53:01.967940675 +0000 UTC m=+1751.743927504" Nov 01 22:53:01.989919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.989890 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:53:01.992700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:01.992668 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mqmv5"] Nov 01 22:53:02.315793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:02.315706 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:02.317555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:02.317526 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:02.317700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:02.317585 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:02.378502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:02.378469 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3973874-a07c-43dc-ab4f-f81b3ed0c5dd" path="/var/lib/kubelet/pods/a3973874-a07c-43dc-ab4f-f81b3ed0c5dd/volumes" Nov 01 22:53:03.318876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:03.318296 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:03.318876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:03.318359 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:03.898277 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:03.898244 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:53:03.911482 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:03.911451 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:53:04.081374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:04.081339 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 22:53:04.319316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:04.319280 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:04.335464 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:04.335417 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:05.337878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:05.328298 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:05.337878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:05.328366 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:06.328765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.318698 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:06.328765 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.318777 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:06.453984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.448429 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 22:53:06.467344 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.467309 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.470815 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.470782 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 22:53:06.477549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.477519 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-64b196\"" Nov 01 22:53:06.477720 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.477696 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-64b196-dockercfg-fpncx\"" Nov 01 22:53:06.638929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.638457 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7bn\" (UniqueName: \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.638929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.638634 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.786886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.786324 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.786886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.786728 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7bn\" (UniqueName: \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.895936 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.805110 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:06.977875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:06.976741 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7bn\" (UniqueName: \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\") pod \"payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:07.120879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:07.110550 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:07.345612 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:07.345565 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:07.345820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:07.345638 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:08.218946 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.215387 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 22:53:08.659107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.659076 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:08.717363 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.671591 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:08.717363 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.671650 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:08.717363 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.700156 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b48ab8\"/\"test-model-b48ab8-kserve-self-signed-certs\"" Nov 01 22:53:08.717363 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.711473 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b48ab8\"/\"default-dockercfg-qfpq5\"" Nov 01 22:53:08.812987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.775109 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b48ab8\"/\"kube-root-ca.crt\"" Nov 01 22:53:08.812987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.775799 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b48ab8\"/\"openshift-service-ca.crt\"" Nov 01 22:53:08.945067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:08.945014 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 22:53:09.049132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.049085 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 22:53:09.273683 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.272960 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" event={"ID":"c2db5323-de4f-433b-aefe-d5a816f5a0eb","Type":"ContainerStarted","Data":"23798d488f8a6e5b49ef882db02074ecb9e6c87ae6767402566fd3901dc318e8"} Nov 01 22:53:09.314999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.314965 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.315179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.315039 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsn8q\" (UniqueName: \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.315407 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.315390 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.315510 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.315492 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.315615 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.315601 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.315689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.315657 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.318594 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.318564 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:53:09.318796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.318767 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 22:53:09.441900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.429145 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:53:09.443757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.443720 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.443993 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.443971 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsn8q\" (UniqueName: \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.444304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.444283 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.444501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.444483 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.444683 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.444665 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.444837 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.444820 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.454873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.444393 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.473582 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.473554 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.492228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.488588 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.492228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.490043 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsn8q\" (UniqueName: \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.492228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.490121 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.494099 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.493830 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\") pod \"test-model-b48ab8-kserve-7b9555c865-f4hjg\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:09.787971 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:09.787937 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:10.287209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:10.287077 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" event={"ID":"c2db5323-de4f-433b-aefe-d5a816f5a0eb","Type":"ContainerStarted","Data":"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b"} Nov 01 22:53:10.287209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:10.287181 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:10.399314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:10.399210 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" podStartSLOduration=4.399173856 podStartE2EDuration="4.399173856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:10.313323334 +0000 UTC m=+1760.089310165" watchObservedRunningTime="2026-11-01 22:53:10.399173856 +0000 UTC m=+1760.175160686" Nov 01 22:53:10.525873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:10.516620 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 22:53:10.591098 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:53:10.590279 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe9edee4_af5d_467f_b2c0_340517f44925.slice/crio-ed2dc679539d0777aaa7d37e5f08e1ebbe620f71019a5cf473fbd4c3c2512139 WatchSource:0}: Error finding container ed2dc679539d0777aaa7d37e5f08e1ebbe620f71019a5cf473fbd4c3c2512139: Status 404 returned error can't find the container with id ed2dc679539d0777aaa7d37e5f08e1ebbe620f71019a5cf473fbd4c3c2512139 Nov 01 22:53:11.132248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.130175 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:11.132248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.132073 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:11.143839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.143812 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:53:11.309874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.309243 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" event={"ID":"fe9edee4-af5d-467f-b2c0-340517f44925","Type":"ContainerStarted","Data":"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64"} Nov 01 22:53:11.309874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.309318 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" event={"ID":"fe9edee4-af5d-467f-b2c0-340517f44925","Type":"ContainerStarted","Data":"ed2dc679539d0777aaa7d37e5f08e1ebbe620f71019a5cf473fbd4c3c2512139"} Nov 01 22:53:11.351246 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.348122 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" podStartSLOduration=3.348098435 podStartE2EDuration="3.348098435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:11.332891309 +0000 UTC m=+1761.108878139" watchObservedRunningTime="2026-11-01 22:53:11.348098435 +0000 UTC m=+1761.124085266" Nov 01 22:53:11.789174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:11.789134 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:17.962104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:17.962056 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:53:17.962506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:17.962137 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:53:18.784019 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:53:18.783980 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:53:19.431416 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.431379 5778 generic.go:356] "Generic (PLEG): container finished" podID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerID="6263b3139ea057744b3e4e1d5371cd59f33b68874942cf9abf3d2d266f75d191" exitCode=0 Nov 01 22:53:19.431906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.431446 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerDied","Data":"6263b3139ea057744b3e4e1d5371cd59f33b68874942cf9abf3d2d266f75d191"} Nov 01 22:53:19.432082 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.432064 5778 scope.go:122] "RemoveContainer" containerID="6263b3139ea057744b3e4e1d5371cd59f33b68874942cf9abf3d2d266f75d191" Nov 01 22:53:19.434374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.434346 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:19.434447 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.434415 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:19.434484 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.434447 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:53:19.789103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.788998 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:19.857199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:19.856825 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:20.464496 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:20.464055 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerStarted","Data":"15cc74f02f1e8212d631e91b21c5de8a7cf2f0d3cdb8742e8a68ffe338d2217e"} Nov 01 22:53:20.476197 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:20.476167 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 22:53:20.644874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:20.624429 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:53:20.645316 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:20.626058 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:20.645549 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:20.645518 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:21.329920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:21.329426 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 22:53:21.624941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:21.624299 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:21.624941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:21.624364 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:22.640365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:22.640327 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:22.641070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:22.641025 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:23.635558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:23.635504 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:23.635889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:23.635819 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:24.636879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:24.636825 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:24.637961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:24.637923 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:25.623267 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:25.623227 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:53:25.623583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:25.623547 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 22:53:26.682887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:26.682464 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:53:26.685291 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:26.685256 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:53:26.714305 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:26.711607 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 22:53:48.778148 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:53:48.778111 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:53:50.722428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:50.722397 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:53:50.732762 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:53:50.732726 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:54:10.141181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.141089 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf"] Nov 01 22:54:10.146376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.146332 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.151036 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.151009 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:10.151276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.151097 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:54:10.153331 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.153306 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf"] Nov 01 22:54:10.173262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.173224 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.173436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.173309 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8gz\" (UniqueName: \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.173611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.173541 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.274682 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.274644 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.274889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.274702 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8gz\" (UniqueName: \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.274942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.274896 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.275376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.275357 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.277706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.277683 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.283764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.283739 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8gz\" (UniqueName: \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\") pod \"maas-api-revoke-sub-ab5dca57-dxmsf\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.469702 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.469663 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:10.781944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:10.781872 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf"] Nov 01 22:54:10.783968 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:54:10.783924 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274be1a9_ad76_479a_8dca_d9bc815c0a17.slice/crio-b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347 WatchSource:0}: Error finding container b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347: Status 404 returned error can't find the container with id b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347 Nov 01 22:54:11.226202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:11.226167 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" event={"ID":"274be1a9-ad76-479a-8dca-d9bc815c0a17","Type":"ContainerStarted","Data":"b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347"} Nov 01 22:54:12.237726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:12.237639 5778 generic.go:356] "Generic (PLEG): container finished" podID="274be1a9-ad76-479a-8dca-d9bc815c0a17" containerID="1af22d377f56239d074598613a8eb7d222b4a183203b2e5fd2ec222f07ca7011" exitCode=0 Nov 01 22:54:12.237726 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:12.237690 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" event={"ID":"274be1a9-ad76-479a-8dca-d9bc815c0a17","Type":"ContainerDied","Data":"1af22d377f56239d074598613a8eb7d222b4a183203b2e5fd2ec222f07ca7011"} Nov 01 22:54:13.547786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.547752 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:13.687394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.687359 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\") pod \"274be1a9-ad76-479a-8dca-d9bc815c0a17\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " Nov 01 22:54:13.687665 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.687650 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\") pod \"274be1a9-ad76-479a-8dca-d9bc815c0a17\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " Nov 01 22:54:13.687729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.687695 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca" pod "274be1a9-ad76-479a-8dca-d9bc815c0a17" (UID: "274be1a9-ad76-479a-8dca-d9bc815c0a17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:13.687768 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.687753 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\" (UniqueName: \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\") pod \"274be1a9-ad76-479a-8dca-d9bc815c0a17\" (UID: \"274be1a9-ad76-479a-8dca-d9bc815c0a17\") " Nov 01 22:54:13.688454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.688429 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:13.690241 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.690208 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca" pod "274be1a9-ad76-479a-8dca-d9bc815c0a17" (UID: "274be1a9-ad76-479a-8dca-d9bc815c0a17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:13.690368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.690262 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz" pod "274be1a9-ad76-479a-8dca-d9bc815c0a17" (UID: "274be1a9-ad76-479a-8dca-d9bc815c0a17"). InnerVolumeSpecName "kube-api-access-hm8gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:13.790053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.789964 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be1a9-ad76-479a-8dca-d9bc815c0a17-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:13.790053 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:13.789996 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm8gz\" (UniqueName: \"kubernetes.io/projected/274be1a9-ad76-479a-8dca-d9bc815c0a17-kube-api-access-hm8gz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:14.260912 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:14.260875 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" event={"ID":"274be1a9-ad76-479a-8dca-d9bc815c0a17","Type":"ContainerDied","Data":"b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347"} Nov 01 22:54:14.261129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:14.260934 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67f9a4138e4756b8230b5e429c1dbced61b6bfea8ba0c4d96065cfad91a2347" Nov 01 22:54:14.261129 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:14.260961 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf" Nov 01 22:54:18.777697 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:54:18.777646 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:54:20.707498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:20.707456 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:54:21.238896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.238830 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94"] Nov 01 22:54:21.240102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.240077 5778 cpu_manager.go:414] "containerMap: removing container" podUID="274be1a9-ad76-479a-8dca-d9bc815c0a17" containerName="revoke-keys" Nov 01 22:54:21.240184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.240104 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274be1a9-ad76-479a-8dca-d9bc815c0a17" containerName="revoke-keys" Nov 01 22:54:21.240365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.240351 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="274be1a9-ad76-479a-8dca-d9bc815c0a17" containerName="revoke-keys" Nov 01 22:54:21.245288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.245259 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.251569 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.251541 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94"] Nov 01 22:54:21.254141 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.254118 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:21.255292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.254117 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:54:21.307392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.307350 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.307673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.307645 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbmf\" (UniqueName: \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.308011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.307986 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.409255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.409213 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbmf\" (UniqueName: \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.409457 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.409410 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.409898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.409569 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.410477 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.410452 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.412811 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.412787 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.420457 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.420426 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbmf\" (UniqueName: \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\") pod \"maas-api-revoke-sub-0b5f6554-l4n94\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.569274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.569183 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:21.876950 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:54:21.875015 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8b1eaa_3d6b_40b1_aecb_bd2ef376465c.slice/crio-73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724 WatchSource:0}: Error finding container 73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724: Status 404 returned error can't find the container with id 73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724 Nov 01 22:54:21.877733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:21.877709 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94"] Nov 01 22:54:22.384737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:22.384639 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" event={"ID":"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c","Type":"ContainerStarted","Data":"123fd3a27ae88bce05c174048b04aaceee210a455dac55bca74ae32939a607a7"} Nov 01 22:54:22.384737 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:22.384722 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" event={"ID":"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c","Type":"ContainerStarted","Data":"73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724"} Nov 01 22:54:22.400554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:22.400501 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" podStartSLOduration=1.400485583 podStartE2EDuration="1.400485583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:22.395392309 +0000 UTC m=+1832.171379137" watchObservedRunningTime="2026-11-01 22:54:22.400485583 +0000 UTC m=+1832.176472412" Nov 01 22:54:23.422413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:23.421988 5778 generic.go:356] "Generic (PLEG): container finished" podID="fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" containerID="123fd3a27ae88bce05c174048b04aaceee210a455dac55bca74ae32939a607a7" exitCode=0 Nov 01 22:54:23.422413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:23.422162 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" event={"ID":"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c","Type":"ContainerDied","Data":"123fd3a27ae88bce05c174048b04aaceee210a455dac55bca74ae32939a607a7"} Nov 01 22:54:24.666397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.666365 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:24.796799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.796684 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\" (UniqueName: \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\") pod \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " Nov 01 22:54:24.803506 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.803445 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\") pod \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " Nov 01 22:54:24.805514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.805479 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\") pod \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\" (UID: \"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c\") " Nov 01 22:54:24.826643 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.826605 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca" pod "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" (UID: "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:24.848405 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.848363 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca" pod "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" (UID: "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:24.850539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.850057 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf" pod "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" (UID: "fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c"). InnerVolumeSpecName "kube-api-access-dgbmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:24.909174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.909138 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbmf\" (UniqueName: \"kubernetes.io/projected/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-kube-api-access-dgbmf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:24.909174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.909172 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:24.909378 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:24.909183 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:25.573292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:25.573249 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" event={"ID":"fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c","Type":"ContainerDied","Data":"73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724"} Nov 01 22:54:25.573606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:25.573582 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94" Nov 01 22:54:25.574270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:25.573596 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e0672e349d44c60ee2ff1c2c1b8131785c01ace281e74c6eb867b24e7f2724" Nov 01 22:54:33.082878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.082624 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:54:33.479223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.479191 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn"] Nov 01 22:54:33.480476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.480453 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" containerName="revoke-keys" Nov 01 22:54:33.480658 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.480642 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" containerName="revoke-keys" Nov 01 22:54:33.481022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.481005 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" containerName="revoke-keys" Nov 01 22:54:33.490222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.490202 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.494258 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.493629 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn"] Nov 01 22:54:33.497473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.496296 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:33.497473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.496413 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:54:33.534114 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.534075 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.534339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.534250 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.534339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.534302 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.636038 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.635999 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.636290 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.636231 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.636380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.636296 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.637024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.636982 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.640774 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.639512 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.646645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.646618 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\") pod \"maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:33.812730 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:33.812646 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:34.114969 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:34.114880 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn"] Nov 01 22:54:34.130165 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:54:34.130128 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d56051f_87d1_4e3f_8e9a_6c18ca69ee71.slice/crio-a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e WatchSource:0}: Error finding container a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e: Status 404 returned error can't find the container with id a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e Nov 01 22:54:34.741937 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:34.741898 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" event={"ID":"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71","Type":"ContainerStarted","Data":"3597b5c3495f53162b447a173443de0538490088d4b6e56ee7fa0100d79fb1df"} Nov 01 22:54:34.742094 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:34.741960 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" event={"ID":"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71","Type":"ContainerStarted","Data":"a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e"} Nov 01 22:54:34.760156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:34.760051 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" podStartSLOduration=1.76002642 podStartE2EDuration="1.76002642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:34.759323148 +0000 UTC m=+1844.535309979" watchObservedRunningTime="2026-11-01 22:54:34.76002642 +0000 UTC m=+1844.536013250" Nov 01 22:54:35.754087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:35.754049 5778 generic.go:356] "Generic (PLEG): container finished" podID="4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" containerID="3597b5c3495f53162b447a173443de0538490088d4b6e56ee7fa0100d79fb1df" exitCode=0 Nov 01 22:54:35.754603 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:35.754135 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" event={"ID":"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71","Type":"ContainerDied","Data":"3597b5c3495f53162b447a173443de0538490088d4b6e56ee7fa0100d79fb1df"} Nov 01 22:54:35.772752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:35.772713 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" containerID="cri-o://e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29" gracePeriod=45 Nov 01 22:54:36.133460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.133429 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:54:36.324990 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.324959 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325183 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325092 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325183 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325170 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325213 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325251 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325498 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\" (UniqueName: \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\") pod \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\" (UID: \"4fcb0993-b3bc-4c5c-b521-13be98647bcb\") " Nov 01 22:54:36.325756 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325621 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:36.326081 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.325823 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:36.326188 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.326080 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:36.326458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.326438 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.326528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.326465 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.326528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.326481 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.327968 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.327941 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:36.328084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.328014 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:36.328353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.328321 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7" pod "4fcb0993-b3bc-4c5c-b521-13be98647bcb" (UID: "4fcb0993-b3bc-4c5c-b521-13be98647bcb"). InnerVolumeSpecName "kube-api-access-nrdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:36.429399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.429359 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdd7\" (UniqueName: \"kubernetes.io/projected/4fcb0993-b3bc-4c5c-b521-13be98647bcb-kube-api-access-nrdd7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.429399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.429402 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcb0993-b3bc-4c5c-b521-13be98647bcb-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.429666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.429420 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcb0993-b3bc-4c5c-b521-13be98647bcb-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:36.765689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.765652 5778 generic.go:356] "Generic (PLEG): container finished" podID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerID="e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29" exitCode=0 Nov 01 22:54:36.766185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.765690 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" event={"ID":"4fcb0993-b3bc-4c5c-b521-13be98647bcb","Type":"ContainerDied","Data":"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29"} Nov 01 22:54:36.766185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.765741 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" event={"ID":"4fcb0993-b3bc-4c5c-b521-13be98647bcb","Type":"ContainerDied","Data":"b08b2f3ca21579e2676a82716561785f587f728be4c7f0333970d78a3217d81b"} Nov 01 22:54:36.766185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.765761 5778 scope.go:122] "RemoveContainer" containerID="e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29" Nov 01 22:54:36.766185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.765701 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc" Nov 01 22:54:36.779271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.779135 5778 scope.go:122] "RemoveContainer" containerID="e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29" Nov 01 22:54:36.779555 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:54:36.779529 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29\": container with ID starting with e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29 not found: ID does not exist" containerID="e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29" Nov 01 22:54:36.779656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.779565 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29"} err="failed to get container status \"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29\": rpc error: code = NotFound desc = could not find container \"e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29\": container with ID starting with e7321b5c6e03d815622e937d0ac1d10e2ab7730822df8b1616b01e7508287a29 not found: ID does not exist" Nov 01 22:54:36.791112 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.790375 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:54:36.795880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.795771 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1799f8/test-model-1799f8-kserve-5d58c8c895-skvqc"] Nov 01 22:54:36.993536 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:36.993506 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:37.082547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.082446 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\") pod \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " Nov 01 22:54:37.082547 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.082518 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\") pod \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " Nov 01 22:54:37.082896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.082873 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\") pod \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\" (UID: \"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71\") " Nov 01 22:54:37.083302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.083271 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca" pod "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" (UID: "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:37.083903 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.083836 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:37.085820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.085702 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l" pod "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" (UID: "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71"). InnerVolumeSpecName "kube-api-access-4k84l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:37.086513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.086430 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca" pod "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" (UID: "4d56051f-87d1-4e3f-8e9a-6c18ca69ee71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:37.185382 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.185347 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-kube-api-access-4k84l\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:37.185555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.185380 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:37.776701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.776661 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" Nov 01 22:54:37.777205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.776659 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn" event={"ID":"4d56051f-87d1-4e3f-8e9a-6c18ca69ee71","Type":"ContainerDied","Data":"a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e"} Nov 01 22:54:37.777205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:37.776798 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e941ff11496fe102c9b9e0ce9581cd4dff1c70bc9179742fdb1b519ec49c2e" Nov 01 22:54:38.377140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:38.377101 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" path="/var/lib/kubelet/pods/4fcb0993-b3bc-4c5c-b521-13be98647bcb/volumes" Nov 01 22:54:48.141807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.131399 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" containerID="cri-o://e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af" gracePeriod=45 Nov 01 22:54:48.488939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.488909 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:54:48.598575 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.598539 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\" (UniqueName: \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.598789 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.598747 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.598895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.598821 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.599025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.598973 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.599102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.599026 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.599102 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.599090 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\") pod \"9d350e26-8da2-4621-b04d-190984552906\" (UID: \"9d350e26-8da2-4621-b04d-190984552906\") " Nov 01 22:54:48.599219 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.599154 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:48.599279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.599257 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:48.599343 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.599289 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:48.600140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.600072 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.600140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.600102 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.600140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.600116 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.602645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.602313 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:48.602645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.602593 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:48.603263 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.603124 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg" pod "9d350e26-8da2-4621-b04d-190984552906" (UID: "9d350e26-8da2-4621-b04d-190984552906"). InnerVolumeSpecName "kube-api-access-8ltqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:48.701926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.701817 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltqg\" (UniqueName: \"kubernetes.io/projected/9d350e26-8da2-4621-b04d-190984552906-kube-api-access-8ltqg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.701926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.701863 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d350e26-8da2-4621-b04d-190984552906-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.701926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.701878 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d350e26-8da2-4621-b04d-190984552906-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:48.779436 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:54:48.779397 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:54:48.920785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.920746 5778 generic.go:356] "Generic (PLEG): container finished" podID="9d350e26-8da2-4621-b04d-190984552906" containerID="e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af" exitCode=0 Nov 01 22:54:48.921039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.920916 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" event={"ID":"9d350e26-8da2-4621-b04d-190984552906","Type":"ContainerDied","Data":"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af"} Nov 01 22:54:48.921039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.920947 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" event={"ID":"9d350e26-8da2-4621-b04d-190984552906","Type":"ContainerDied","Data":"cd6a2dcf9cc100ad4151cdee9ecf817308aefe890c0649b20a32c405910a2caa"} Nov 01 22:54:48.921039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.920964 5778 scope.go:122] "RemoveContainer" containerID="e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af" Nov 01 22:54:48.921212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.921089 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9" Nov 01 22:54:48.939571 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.939545 5778 scope.go:122] "RemoveContainer" containerID="e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af" Nov 01 22:54:48.939922 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:54:48.939891 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af\": container with ID starting with e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af not found: ID does not exist" containerID="e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af" Nov 01 22:54:48.940059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.939932 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af"} err="failed to get container status \"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af\": rpc error: code = NotFound desc = could not find container \"e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af\": container with ID starting with e0792e2929fe385cdc19d942e1e848768dc317f90776b3f4b5a6c5656571a8af not found: ID does not exist" Nov 01 22:54:48.955299 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.955215 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:54:48.963334 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:48.963297 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-44a9aa/test-model-44a9aa-kserve-7d96b84dc-n9jm9"] Nov 01 22:54:50.043206 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:50.043168 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn"] Nov 01 22:54:50.108608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:50.106097 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1799f8-lb7bn"] Nov 01 22:54:50.379126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:50.379050 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" path="/var/lib/kubelet/pods/4d56051f-87d1-4e3f-8e9a-6c18ca69ee71/volumes" Nov 01 22:54:50.379544 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:50.379527 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d350e26-8da2-4621-b04d-190984552906" path="/var/lib/kubelet/pods/9d350e26-8da2-4621-b04d-190984552906/volumes" Nov 01 22:54:57.873608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:57.873512 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:54:57.874781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:57.874594 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" containerID="cri-o://3ebe687db2b09e0c25bcfba6f9441a9222408e47b6a4a3e0217d79c3dea68a74" gracePeriod=30 Nov 01 22:54:58.404507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.404470 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr"] Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405302 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" containerName="revoke-keys" Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405322 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" containerName="revoke-keys" Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405343 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405348 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405360 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" Nov 01 22:54:58.405322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405364 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" Nov 01 22:54:58.405807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405515 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d56051f-87d1-4e3f-8e9a-6c18ca69ee71" containerName="revoke-keys" Nov 01 22:54:58.405807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405526 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcb0993-b3bc-4c5c-b521-13be98647bcb" containerName="main" Nov 01 22:54:58.405807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.405534 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d350e26-8da2-4621-b04d-190984552906" containerName="main" Nov 01 22:54:58.411843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.411817 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.419829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.419796 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr"] Nov 01 22:54:58.478635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.478587 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbzk\" (UniqueName: \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.478845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.478651 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.478845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.478678 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.493206 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.493170 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:58.493206 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.493179 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:54:58.620891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.617067 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbzk\" (UniqueName: \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.620891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.617135 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.620891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.617175 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.624874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.623662 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.659460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.659250 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbzk\" (UniqueName: \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.681917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.680512 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:58.744315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:58.744280 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:54:59.040010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:59.039977 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr"] Nov 01 22:54:59.041214 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:54:59.041185 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e179e6a_e7eb_42b0_b8c2_64421249e18d.slice/crio-b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d WatchSource:0}: Error finding container b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d: Status 404 returned error can't find the container with id b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d Nov 01 22:54:59.403240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:54:59.401231 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" event={"ID":"4e179e6a-e7eb-42b0-b8c2-64421249e18d","Type":"ContainerStarted","Data":"b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d"} Nov 01 22:55:00.411468 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:00.411436 5778 generic.go:356] "Generic (PLEG): container finished" podID="4e179e6a-e7eb-42b0-b8c2-64421249e18d" containerID="9131eee833681fe590825f8178d5ac1b76d377b74121cf03d47b8d99f6ce361a" exitCode=0 Nov 01 22:55:00.411906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:00.411527 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" event={"ID":"4e179e6a-e7eb-42b0-b8c2-64421249e18d","Type":"ContainerDied","Data":"9131eee833681fe590825f8178d5ac1b76d377b74121cf03d47b8d99f6ce361a"} Nov 01 22:55:01.637151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.637125 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:55:01.799698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.799611 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\") pod \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " Nov 01 22:55:01.799902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.799877 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\" (UniqueName: \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\") pod \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " Nov 01 22:55:01.799972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.799952 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\") pod \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\" (UID: \"4e179e6a-e7eb-42b0-b8c2-64421249e18d\") " Nov 01 22:55:01.800019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.799961 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca" pod "4e179e6a-e7eb-42b0-b8c2-64421249e18d" (UID: "4e179e6a-e7eb-42b0-b8c2-64421249e18d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:01.800686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.800666 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:01.802247 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.802224 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk" pod "4e179e6a-e7eb-42b0-b8c2-64421249e18d" (UID: "4e179e6a-e7eb-42b0-b8c2-64421249e18d"). InnerVolumeSpecName "kube-api-access-8hbzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:01.802339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.802262 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca" pod "4e179e6a-e7eb-42b0-b8c2-64421249e18d" (UID: "4e179e6a-e7eb-42b0-b8c2-64421249e18d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:01.902626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.902586 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbzk\" (UniqueName: \"kubernetes.io/projected/4e179e6a-e7eb-42b0-b8c2-64421249e18d-kube-api-access-8hbzk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:01.902626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:01.902622 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e179e6a-e7eb-42b0-b8c2-64421249e18d-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:02.436097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:02.436066 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" Nov 01 22:55:02.436097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:02.436072 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr" event={"ID":"4e179e6a-e7eb-42b0-b8c2-64421249e18d","Type":"ContainerDied","Data":"b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d"} Nov 01 22:55:02.436376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:02.436129 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b49ac017c0e6e2fe31d3904a45763267342254818e4eeab5b234b5f711826b5d" Nov 01 22:55:03.450381 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.450346 5778 generic.go:356] "Generic (PLEG): container finished" podID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerID="3ebe687db2b09e0c25bcfba6f9441a9222408e47b6a4a3e0217d79c3dea68a74" exitCode=0 Nov 01 22:55:03.450779 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.450405 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" event={"ID":"eb98f5d2-0a05-479a-887a-f504e20bcd89","Type":"ContainerDied","Data":"3ebe687db2b09e0c25bcfba6f9441a9222408e47b6a4a3e0217d79c3dea68a74"} Nov 01 22:55:03.644657 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.644624 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:55:03.650415 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650382 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.650579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650454 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.650579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650545 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.650715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650587 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.650715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650658 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.650715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650680 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650774 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650830 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650955 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.650996 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.651036 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\") pod \"eb98f5d2-0a05-479a-887a-f504e20bcd89\" (UID: \"eb98f5d2-0a05-479a-887a-f504e20bcd89\") " Nov 01 22:55:03.651179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.651044 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.651865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.651351 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:03.651865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.651435 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.652265 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.652244 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.652358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.652274 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.652358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.652289 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.652358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.652304 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.652358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.652317 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb98f5d2-0a05-479a-887a-f504e20bcd89-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.654103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.654078 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:03.654239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.654220 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "kube-api-access-c4nc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.654343 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.654320 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.654669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.654645 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token" pod "eb98f5d2-0a05-479a-887a-f504e20bcd89" (UID: "eb98f5d2-0a05-479a-887a-f504e20bcd89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.718802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.715819 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753155 5778 cpu_manager.go:414] "containerMap: removing container" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753184 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753205 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4e179e6a-e7eb-42b0-b8c2-64421249e18d" containerName="revoke-keys" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753213 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e179e6a-e7eb-42b0-b8c2-64421249e18d" containerName="revoke-keys" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753391 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" containerName="istio-proxy" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.753405 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e179e6a-e7eb-42b0-b8c2-64421249e18d" containerName="revoke-keys" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.754198 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nc8\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-kube-api-access-c4nc8\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.754226 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.754240 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.759806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.754254 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb98f5d2-0a05-479a-887a-f504e20bcd89-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.767911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.764715 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 22:55:03.767911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.764841 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855090 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855054 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855135 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855279 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855250 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855291 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855369 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855353 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855438 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855490 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855465 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hc6\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855529 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855498 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.855685 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.855671 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.956983 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957105 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957179 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957254 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957284 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957442 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957524 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957539 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957564 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hc6\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.957649 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957629 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.958023 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.957671 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.958314 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.958289 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.958427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.958404 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.959981 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.959950 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.977273 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.961360 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.990698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.983553 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:03.990698 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:03.983691 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hc6\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:04.109140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.109105 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:04.440094 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.440060 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 22:55:04.514877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.514808 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:55:04.515489 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.515457 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:55:04.515634 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.515618 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:55:04.545203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.544082 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" Nov 01 22:55:04.545203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.544090 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9" event={"ID":"eb98f5d2-0a05-479a-887a-f504e20bcd89","Type":"ContainerDied","Data":"c2d7acafcbb5a902209e568d6d8064d49dffee74d6e020ac0741108460abdf2c"} Nov 01 22:55:04.545203 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.544159 5778 scope.go:122] "RemoveContainer" containerID="3ebe687db2b09e0c25bcfba6f9441a9222408e47b6a4a3e0217d79c3dea68a74" Nov 01 22:55:04.554509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.554455 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" event={"ID":"56f77ba7-e65c-4ac6-9478-3cd83b6567d8","Type":"ContainerStarted","Data":"ddc9c1ad8fcf3fe0e804e0e9f5bbfcf92990ec455efb451b5bcc6d286dae3dce"} Nov 01 22:55:04.581447 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.581416 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:55:04.586606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:04.586578 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-qmkv9"] Nov 01 22:55:05.628293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:05.628250 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" event={"ID":"56f77ba7-e65c-4ac6-9478-3cd83b6567d8","Type":"ContainerStarted","Data":"2b5ba7cb2f1f1f0588af95409ab1d8c172a7b6cb1bc0af4dde0623c9ffda31c5"} Nov 01 22:55:05.680212 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:05.680140 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podStartSLOduration=69.68011804 podStartE2EDuration="1m9.68011804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:05.665349706 +0000 UTC m=+1875.441336549" watchObservedRunningTime="2026-11-01 22:55:05.68011804 +0000 UTC m=+1875.456104871" Nov 01 22:55:06.110124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:06.110085 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:06.111589 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:06.111556 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:06.111743 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:06.111621 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:06.379412 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:06.379320 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb98f5d2-0a05-479a-887a-f504e20bcd89" path="/var/lib/kubelet/pods/eb98f5d2-0a05-479a-887a-f504e20bcd89/volumes" Nov 01 22:55:07.109645 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:07.109604 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:07.110099 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:07.109673 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:08.109963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:08.109926 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:08.110497 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:08.109994 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:09.115874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:09.115033 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:09.115874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:09.115104 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:10.110182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:10.110144 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:10.110394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:10.110206 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:11.109722 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:11.109685 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:55:11.110155 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:11.109753 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 22:55:12.114838 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:12.114803 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:12.115408 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:12.115381 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:12.116221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:12.116200 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 22:55:14.774182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:14.774143 5778 generic.go:356] "Generic (PLEG): container finished" podID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerID="23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377" exitCode=137 Nov 01 22:55:14.774664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:14.774228 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerDied","Data":"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377"} Nov 01 22:55:14.774831 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:14.774812 5778 scope.go:122] "RemoveContainer" containerID="23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377" Nov 01 22:55:14.837608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:14.837576 5778 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:55:15.787296 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:15.787236 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerStarted","Data":"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66"} Nov 01 22:55:15.787729 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:15.787329 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:55:17.213878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:17.207910 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr"] Nov 01 22:55:17.234878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:17.232308 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44a9aa-6zwtr"] Nov 01 22:55:18.379294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:18.379256 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e179e6a-e7eb-42b0-b8c2-64421249e18d" path="/var/lib/kubelet/pods/4e179e6a-e7eb-42b0-b8c2-64421249e18d/volumes" Nov 01 22:55:18.775591 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:55:18.775556 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:55:22.071174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:22.071119 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:55:23.834895 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:23.834797 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:55:26.808816 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:26.803559 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 22:55:34.065595 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:34.065508 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:55:37.115520 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:37.115444 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" containerID="cri-o://2a6a03e056f1236383b9cb14dfa4e271c84a4ae97a34658daae10f3808d00b1b" gracePeriod=45 Nov 01 22:55:37.278001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:37.277960 5778 generic.go:356] "Generic (PLEG): container finished" podID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerID="2a6a03e056f1236383b9cb14dfa4e271c84a4ae97a34658daae10f3808d00b1b" exitCode=0 Nov 01 22:55:37.278193 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:37.278033 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" event={"ID":"eab12b5d-fabd-4ac3-a300-f735dcc791ff","Type":"ContainerDied","Data":"2a6a03e056f1236383b9cb14dfa4e271c84a4ae97a34658daae10f3808d00b1b"} Nov 01 22:55:37.379098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:37.379006 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:55:37.889709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:37.889676 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:55:38.036967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.036914 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.037201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037026 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\" (UniqueName: \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.037201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037128 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.037201 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037171 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:38.037373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037285 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.037373 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037317 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.037475 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037384 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\") pod \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\" (UID: \"eab12b5d-fabd-4ac3-a300-f735dcc791ff\") " Nov 01 22:55:38.038014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.037988 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:38.038222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.038094 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:38.038430 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.038408 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.038498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.038438 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.038498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.038453 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.039901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.039864 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "kube-api-access-fjlhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:38.040085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.040055 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:38.040085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.040078 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs" pod "eab12b5d-fabd-4ac3-a300-f735dcc791ff" (UID: "eab12b5d-fabd-4ac3-a300-f735dcc791ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:38.140410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.140313 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab12b5d-fabd-4ac3-a300-f735dcc791ff-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.140410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.140347 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjlhk\" (UniqueName: \"kubernetes.io/projected/eab12b5d-fabd-4ac3-a300-f735dcc791ff-kube-api-access-fjlhk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.140410 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.140357 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab12b5d-fabd-4ac3-a300-f735dcc791ff-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.289953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.289904 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" Nov 01 22:55:38.289953 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.289933 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" event={"ID":"eab12b5d-fabd-4ac3-a300-f735dcc791ff","Type":"ContainerDied","Data":"ef81ab40034f582aec545b2588cc96f480d8ff4d8d99bd352d9b67d30de33e5d"} Nov 01 22:55:38.290171 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.289989 5778 scope.go:122] "RemoveContainer" containerID="2a6a03e056f1236383b9cb14dfa4e271c84a4ae97a34658daae10f3808d00b1b" Nov 01 22:55:38.314922 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.314889 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:55:38.319772 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.319741 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf"] Nov 01 22:55:38.379320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.379286 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" path="/var/lib/kubelet/pods/eab12b5d-fabd-4ac3-a300-f735dcc791ff/volumes" Nov 01 22:55:38.866235 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.866183 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" containerID="cri-o://7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a" gracePeriod=45 Nov 01 22:55:38.875441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:38.875402 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e202e7/test-model-e202e7-kserve-c96679b4f-tf2sf" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": context deadline exceeded" Nov 01 22:55:39.217283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.216932 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:55:39.307358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.307320 5778 generic.go:356] "Generic (PLEG): container finished" podID="42389da7-468d-4a61-9942-c67def02d4d3" containerID="7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a" exitCode=0 Nov 01 22:55:39.307583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.307378 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" Nov 01 22:55:39.307583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.307408 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" event={"ID":"42389da7-468d-4a61-9942-c67def02d4d3","Type":"ContainerDied","Data":"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a"} Nov 01 22:55:39.307583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.307460 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll" event={"ID":"42389da7-468d-4a61-9942-c67def02d4d3","Type":"ContainerDied","Data":"a5321a3a97e9917654175c9bde6019e2605345c5560eb9e2c1fe67295234d990"} Nov 01 22:55:39.307583 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.307483 5778 scope.go:122] "RemoveContainer" containerID="7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a" Nov 01 22:55:39.345024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.345000 5778 scope.go:122] "RemoveContainer" containerID="7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a" Nov 01 22:55:39.345467 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:55:39.345419 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a\": container with ID starting with 7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a not found: ID does not exist" containerID="7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a" Nov 01 22:55:39.345592 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.345464 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a"} err="failed to get container status \"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a\": rpc error: code = NotFound desc = could not find container \"7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a\": container with ID starting with 7cf11e47a54ebb144d2e3a7f6f6a05a6dfafea18f90cbd492094e0dc588e630a not found: ID does not exist" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378154 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378266 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\" (UniqueName: \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378365 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378426 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378578 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.378665 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\") pod \"42389da7-468d-4a61-9942-c67def02d4d3\" (UID: \"42389da7-468d-4a61-9942-c67def02d4d3\") " Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.379835 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.383567 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.383759 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.384014 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.385514 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "kube-api-access-qdb7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.392313 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.393723 5778 cpu_manager.go:414] "containerMap: removing container" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.393743 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.393783 5778 cpu_manager.go:414] "containerMap: removing container" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.393795 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.394071 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="42389da7-468d-4a61-9942-c67def02d4d3" containerName="main" Nov 01 22:55:39.397877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.394089 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab12b5d-fabd-4ac3-a300-f735dcc791ff" containerName="main" Nov 01 22:55:39.401540 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.400588 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs" pod "42389da7-468d-4a61-9942-c67def02d4d3" (UID: "42389da7-468d-4a61-9942-c67def02d4d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:39.401540 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.401490 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.411126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.410240 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 22:55:39.426342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.426253 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c49240\"/\"default-dockercfg-s67dn\"" Nov 01 22:55:39.426342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.426336 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c49240\"/\"openshift-service-ca.crt\"" Nov 01 22:55:39.426483 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.426424 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c49240\"/\"kube-root-ca.crt\"" Nov 01 22:55:39.426664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.426534 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c49240\"/\"test-model-c49240-kserve-self-signed-certs\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482215 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482249 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482265 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482278 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42389da7-468d-4a61-9942-c67def02d4d3-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482294 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42389da7-468d-4a61-9942-c67def02d4d3-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.488150 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.482311 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdb7g\" (UniqueName: \"kubernetes.io/projected/42389da7-468d-4a61-9942-c67def02d4d3-kube-api-access-qdb7g\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.584396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.584344 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.584606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.584562 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.584686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.584611 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.584876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.584821 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbxh\" (UniqueName: \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.585073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.585016 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.585195 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.585097 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.657948 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.657831 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:55:39.665667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.665610 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-58c957/test-model-58c957-kserve-64d6df7dd5-q42ll"] Nov 01 22:55:39.687425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687323 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbxh\" (UniqueName: \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.687577 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687444 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.687577 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687492 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.687671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687653 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.687806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687788 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.687957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.687833 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.688768 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.688201 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.688768 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.688326 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.688768 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.688737 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.691483 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.691458 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.692342 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.692320 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.695949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.695921 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbxh\" (UniqueName: \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\") pod \"test-model-c49240-kserve-75974c7d6c-8jlw7\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:39.783744 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:39.783443 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:40.345907 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:55:40.345218 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc292b2f5_7ebf_4425_8fbd_6de7f1edb8b0.slice/crio-1c8c72a35e629de05815a9a8050fc03fe7c3d0ba71fc70a311be2e2f40f92be1 WatchSource:0}: Error finding container 1c8c72a35e629de05815a9a8050fc03fe7c3d0ba71fc70a311be2e2f40f92be1: Status 404 returned error can't find the container with id 1c8c72a35e629de05815a9a8050fc03fe7c3d0ba71fc70a311be2e2f40f92be1 Nov 01 22:55:40.346926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:40.346891 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 22:55:40.381806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:40.381767 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42389da7-468d-4a61-9942-c67def02d4d3" path="/var/lib/kubelet/pods/42389da7-468d-4a61-9942-c67def02d4d3/volumes" Nov 01 22:55:41.334957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.334906 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" event={"ID":"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0","Type":"ContainerStarted","Data":"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb"} Nov 01 22:55:41.334957 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.334959 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" event={"ID":"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0","Type":"ContainerStarted","Data":"1c8c72a35e629de05815a9a8050fc03fe7c3d0ba71fc70a311be2e2f40f92be1"} Nov 01 22:55:41.354696 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.354630 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" podStartSLOduration=2.354610926 podStartE2EDuration="2.354610926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:41.350789776 +0000 UTC m=+1911.126776609" watchObservedRunningTime="2026-11-01 22:55:41.354610926 +0000 UTC m=+1911.130597759" Nov 01 22:55:41.536714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.536641 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 22:55:41.543638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.543609 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.553757 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.553703 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64b196\"/\"default-dockercfg-rjjzw\"" Nov 01 22:55:41.554117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.554077 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64b196\"/\"test-model-64b196-kserve-self-signed-certs\"" Nov 01 22:55:41.555113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.555093 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64b196\"/\"kube-root-ca.crt\"" Nov 01 22:55:41.555315 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.555301 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64b196\"/\"openshift-service-ca.crt\"" Nov 01 22:55:41.558395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.558362 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 22:55:41.670295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.670262 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.670502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.670391 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.670502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.670434 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.670879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.670835 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.671124 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.671097 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vqv\" (UniqueName: \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.671232 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.671193 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.777626 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.777588 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.777985 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.777961 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.778143 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.778128 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.782873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.778340 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.787700 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vqv\" (UniqueName: \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.787759 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.785924 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.786217 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.783423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.783667 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.839429 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.839393 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vqv\" (UniqueName: \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.872723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.870973 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.872996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.871420 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\") pod \"test-model-64b196-kserve-65f8cb6c4c-2g876\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:41.887347 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:41.887065 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:42.416551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:42.415070 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 22:55:43.437806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:43.437733 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" event={"ID":"912a564b-059a-4ee5-8abb-0b476a5480c4","Type":"ContainerStarted","Data":"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288"} Nov 01 22:55:43.437806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:43.437802 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" event={"ID":"912a564b-059a-4ee5-8abb-0b476a5480c4","Type":"ContainerStarted","Data":"177cdfcbd2f99905f54ca0dd7b20ccf1e4901c1168b8584da49b8843405ba6da"} Nov 01 22:55:43.487733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:43.487682 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podStartSLOduration=2.487667522 podStartE2EDuration="2.487667522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:43.479628894 +0000 UTC m=+1913.255615722" watchObservedRunningTime="2026-11-01 22:55:43.487667522 +0000 UTC m=+1913.263654350" Nov 01 22:55:43.891264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:43.891187 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:48.791498 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:55:48.791459 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:55:49.109693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.109614 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" containerID="cri-o://76d868d85de89c8a700703a358f5c4a372d1c8e751d60a9834126e7c36054661" gracePeriod=45 Nov 01 22:55:49.514591 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.514561 5778 generic.go:356] "Generic (PLEG): container finished" podID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerID="76d868d85de89c8a700703a358f5c4a372d1c8e751d60a9834126e7c36054661" exitCode=0 Nov 01 22:55:49.514782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.514645 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" event={"ID":"2d9b8603-13ce-4f85-9737-83efaeed82a2","Type":"ContainerDied","Data":"76d868d85de89c8a700703a358f5c4a372d1c8e751d60a9834126e7c36054661"} Nov 01 22:55:49.514782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.514714 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" event={"ID":"2d9b8603-13ce-4f85-9737-83efaeed82a2","Type":"ContainerDied","Data":"9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9"} Nov 01 22:55:49.514782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.514733 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6832eda32d986c4dee4245237aa3fc2ccc413689857a4e5bc5b6570333fce9" Nov 01 22:55:49.533126 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.533100 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:55:49.671407 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671373 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.671606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671541 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.671709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671693 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.671807 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671791 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\" (UniqueName: \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.671878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671845 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.671933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671846 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:49.672002 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.671948 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\") pod \"2d9b8603-13ce-4f85-9737-83efaeed82a2\" (UID: \"2d9b8603-13ce-4f85-9737-83efaeed82a2\") " Nov 01 22:55:49.672271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.672241 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:49.672380 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.672251 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:49.673156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.673128 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.673260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.673164 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.673260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.673182 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.673976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.673957 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:49.674162 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.674148 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:49.674396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.674375 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc" pod "2d9b8603-13ce-4f85-9737-83efaeed82a2" (UID: "2d9b8603-13ce-4f85-9737-83efaeed82a2"). InnerVolumeSpecName "kube-api-access-wq9kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:49.775209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.775124 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9b8603-13ce-4f85-9737-83efaeed82a2-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.775209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.775153 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9kc\" (UniqueName: \"kubernetes.io/projected/2d9b8603-13ce-4f85-9737-83efaeed82a2-kube-api-access-wq9kc\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.775209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.775165 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9b8603-13ce-4f85-9737-83efaeed82a2-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:49.785008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.784976 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:49.798196 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:49.798170 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:50.416175 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:50.416115 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:50.525789 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:50.525756 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd" Nov 01 22:55:50.543516 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:50.543485 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 22:55:50.548185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:50.548160 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:55:50.554721 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:50.554697 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f07902/test-model-f07902-kserve-7547dddd6c-qw7gd"] Nov 01 22:55:51.891395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:51.891337 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:51.905635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:51.905602 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:52.393063 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.393031 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" path="/var/lib/kubelet/pods/2d9b8603-13ce-4f85-9737-83efaeed82a2/volumes" Nov 01 22:55:52.415945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.415904 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" containerID="cri-o://1d58fc300c10e42bde03c18470772a65361db1883ace110887e4c16d4f467c32" gracePeriod=45 Nov 01 22:55:52.572256 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.572216 5778 generic.go:356] "Generic (PLEG): container finished" podID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerID="1d58fc300c10e42bde03c18470772a65361db1883ace110887e4c16d4f467c32" exitCode=0 Nov 01 22:55:52.573514 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.573482 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" event={"ID":"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5","Type":"ContainerDied","Data":"1d58fc300c10e42bde03c18470772a65361db1883ace110887e4c16d4f467c32"} Nov 01 22:55:52.591095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.591063 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:55:52.820843 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.820809 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:55:52.879790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.879749 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\" (UniqueName: \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.879930 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.880129 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.880278 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.880607 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.880673 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881046 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.880723 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\") pod \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\" (UID: \"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5\") " Nov 01 22:55:52.881432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.881086 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:52.881432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.881358 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:52.882236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.882160 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:52.882236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.882187 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:52.882236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.882199 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:52.883709 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.883684 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:52.883839 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.883815 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:52.883935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.883816 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm" pod "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" (UID: "8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5"). InnerVolumeSpecName "kube-api-access-j8kxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:52.996874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.983768 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:52.996874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.983800 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kxm\" (UniqueName: \"kubernetes.io/projected/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-kube-api-access-j8kxm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:52.996874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:52.983811 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:53.541199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.541163 5778 scope.go:122] "RemoveContainer" containerID="76d868d85de89c8a700703a358f5c4a372d1c8e751d60a9834126e7c36054661" Nov 01 22:55:53.561160 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.559209 5778 scope.go:122] "RemoveContainer" containerID="1d58fc300c10e42bde03c18470772a65361db1883ace110887e4c16d4f467c32" Nov 01 22:55:53.585069 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.585043 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" Nov 01 22:55:53.585486 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.585042 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" event={"ID":"8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5","Type":"ContainerDied","Data":"7e6442a732e9c5d7116072eb30b066d1a4b7a8db3b2116720d2895c48a6d8e5e"} Nov 01 22:55:53.617455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.617006 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:55:53.642458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.642340 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt"] Nov 01 22:55:53.694077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:53.694033 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e716d/test-model-0e716d-kserve-8649848597-jzdmt" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:54.443919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:54.443716 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" path="/var/lib/kubelet/pods/8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5/volumes" Nov 01 22:55:55.114724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.112677 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:55:55.114724 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.112953 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" containerID="cri-o://47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64" gracePeriod=30 Nov 01 22:55:55.864312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.864277 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:55:55.864844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.864529 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" containerID="cri-o://bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867" gracePeriod=30 Nov 01 22:55:55.914540 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.914499 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:55:55.923918 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.917317 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" containerID="cri-o://bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1" gracePeriod=30 Nov 01 22:55:55.990108 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.990076 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:55:55.990310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:55.990289 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" containerID="cri-o://75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a" gracePeriod=30 Nov 01 22:55:56.569884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.564913 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:55:56.569884 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.564979 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:55:56.597980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.597939 5778 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:55:56.598167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.598023 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:55:56.709386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.709351 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:55:56.709555 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:56.709424 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:55:58.134462 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:58.134409 5778 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:55:58.134924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:55:58.134504 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:56:00.586841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.586812 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:56:00.665427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.665392 5778 generic.go:356] "Generic (PLEG): container finished" podID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerID="47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64" exitCode=0 Nov 01 22:56:00.665638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.665459 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" Nov 01 22:56:00.665638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.665530 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" event={"ID":"565eebb6-d87e-4f1c-bae0-e78d741850d4","Type":"ContainerDied","Data":"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64"} Nov 01 22:56:00.665638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.665580 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh" event={"ID":"565eebb6-d87e-4f1c-bae0-e78d741850d4","Type":"ContainerDied","Data":"8241e7c98a17eb0a5a1340e955575b9fbdc234f041440669364b6a318bcd8bcb"} Nov 01 22:56:00.665638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.665610 5778 scope.go:122] "RemoveContainer" containerID="47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64" Nov 01 22:56:00.678484 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.678447 5778 scope.go:122] "RemoveContainer" containerID="47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64" Nov 01 22:56:00.678813 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:00.678789 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64\": container with ID starting with 47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64 not found: ID does not exist" containerID="47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64" Nov 01 22:56:00.678945 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.678826 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64"} err="failed to get container status \"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64\": rpc error: code = NotFound desc = could not find container \"47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64\": container with ID starting with 47e3fbae56742e95212a9a96bccba5eed840b29ef2ea7e34aa3746bf87a7df64 not found: ID does not exist" Nov 01 22:56:00.717809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.717762 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718028 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.717843 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718028 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718018 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718085 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718133 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:00.718173 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718145 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718261 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:00.718367 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718314 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718424 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718401 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718631 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718603 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718647 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\") pod \"565eebb6-d87e-4f1c-bae0-e78d741850d4\" (UID: \"565eebb6-d87e-4f1c-bae0-e78d741850d4\") " Nov 01 22:56:00.718867 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718821 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:00.718952 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.718872 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:00.719065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719037 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:00.719776 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719756 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.719927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719785 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.719927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719802 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.719927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719816 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.719927 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.719829 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565eebb6-d87e-4f1c-bae0-e78d741850d4-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.721169 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.721136 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:00.721262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.721197 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:00.721488 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.721464 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:00.721624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.721609 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z" pod "565eebb6-d87e-4f1c-bae0-e78d741850d4" (UID: "565eebb6-d87e-4f1c-bae0-e78d741850d4"). InnerVolumeSpecName "kube-api-access-qkz9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:00.861893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.842450 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.861893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.842490 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.861893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.842505 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565eebb6-d87e-4f1c-bae0-e78d741850d4-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.861893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:00.842520 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkz9z\" (UniqueName: \"kubernetes.io/projected/565eebb6-d87e-4f1c-bae0-e78d741850d4-kube-api-access-qkz9z\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.015458 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.015425 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:56:01.023361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.023323 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-dnjnh"] Nov 01 22:56:01.237732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.237700 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:56:01.333159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.333126 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:56:01.352938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.352903 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353039 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353099 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353191 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353183 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353223 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353377 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353365 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353425 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353619 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353600 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.353705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.353688 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\") pod \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\" (UID: \"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4\") " Nov 01 22:56:01.354796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.354766 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.355368 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.355342 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.355540 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.355520 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.355806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.355785 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:01.356661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.356633 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.371841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.371808 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "kube-api-access-z2hj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.373551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.372399 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.373551 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.372437 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:01.375075 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.375045 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token" pod "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" (UID: "5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.436539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.436506 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:56:01.455656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455579 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.455656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455649 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.455844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455687 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.455844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455767 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.455844 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455811 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.456029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455943 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.456029 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.455994 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.456136 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456049 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:01.456190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456179 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.456290 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456264 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\") pod \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\" (UID: \"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8\") " Nov 01 22:56:01.456421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456310 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.456421 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456332 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.456733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456710 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.456996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.456971 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.457459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457439 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457464 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457479 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457491 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457505 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457519 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457545 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457532 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457546 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hj7\" (UniqueName: \"kubernetes.io/projected/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-kube-api-access-z2hj7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457560 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457573 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457588 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457601 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457616 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.457781 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.457629 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.458558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.458536 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:01.459027 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.459004 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.459151 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.459133 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "kube-api-access-7cq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.460715 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.460693 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token" pod "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" (UID: "5b6ebf20-b0bd-4afe-8a67-de7f357db9f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.558942 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.558909 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559049 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559147 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559112 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559271 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559188 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559279 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559320 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559394 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559411 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.559432 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559425 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559598 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559466 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\") pod \"5369d805-44aa-4cc2-9c18-b93525ab9b61\" (UID: \"5369d805-44aa-4cc2-9c18-b93525ab9b61\") " Nov 01 22:56:01.559663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559627 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.559941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.559918 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560370 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560654 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560676 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560687 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560702 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cq2x\" (UniqueName: \"kubernetes.io/projected/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-kube-api-access-7cq2x\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560711 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5369d805-44aa-4cc2-9c18-b93525ab9b61-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560719 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560727 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.560752 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.560735 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.561338 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.561316 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.563379 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.563351 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:01.563987 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.563961 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.564140 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.564117 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:01.564276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.564252 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk" pod "5369d805-44aa-4cc2-9c18-b93525ab9b61" (UID: "5369d805-44aa-4cc2-9c18-b93525ab9b61"). InnerVolumeSpecName "kube-api-access-6f6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:01.663898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.663485 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.663898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.663522 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.663898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.663535 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.663898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.663549 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5369d805-44aa-4cc2-9c18-b93525ab9b61-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.663898 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.663563 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6jk\" (UniqueName: \"kubernetes.io/projected/5369d805-44aa-4cc2-9c18-b93525ab9b61-kube-api-access-6f6jk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:01.681045 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.681016 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" Nov 01 22:56:01.681276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.681023 5778 generic.go:356] "Generic (PLEG): container finished" podID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerID="bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1" exitCode=0 Nov 01 22:56:01.681276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.681045 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" event={"ID":"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8","Type":"ContainerDied","Data":"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1"} Nov 01 22:56:01.681276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.681171 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85" event={"ID":"5b6ebf20-b0bd-4afe-8a67-de7f357db9f8","Type":"ContainerDied","Data":"20fc53d1ffd380a6f95fcdc61b2978942830e72e0e785d2902e1fd40f6c82e82"} Nov 01 22:56:01.681276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.681205 5778 scope.go:122] "RemoveContainer" containerID="bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1" Nov 01 22:56:01.683132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.683004 5778 generic.go:356] "Generic (PLEG): container finished" podID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerID="75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a" exitCode=0 Nov 01 22:56:01.683132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.683061 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" Nov 01 22:56:01.683132 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.683091 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" event={"ID":"5369d805-44aa-4cc2-9c18-b93525ab9b61","Type":"ContainerDied","Data":"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a"} Nov 01 22:56:01.683586 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.683137 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56" event={"ID":"5369d805-44aa-4cc2-9c18-b93525ab9b61","Type":"ContainerDied","Data":"0b03717823dddc7304cc14ead03e3c9014069064d98b26ceceb201aabee71b2c"} Nov 01 22:56:01.686101 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.686043 5778 generic.go:356] "Generic (PLEG): container finished" podID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerID="bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867" exitCode=0 Nov 01 22:56:01.686205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.686097 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" Nov 01 22:56:01.686205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.686071 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" event={"ID":"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4","Type":"ContainerDied","Data":"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867"} Nov 01 22:56:01.686375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.686201 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf" event={"ID":"5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4","Type":"ContainerDied","Data":"fa53494555338288dc6fd800df9e927a47fa4c9d1d0ab1c25ebf122537ab37f7"} Nov 01 22:56:01.704238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.704208 5778 scope.go:122] "RemoveContainer" containerID="bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1" Nov 01 22:56:01.704607 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:01.704581 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1\": container with ID starting with bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1 not found: ID does not exist" containerID="bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1" Nov 01 22:56:01.704691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.704623 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1"} err="failed to get container status \"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1\": rpc error: code = NotFound desc = could not find container \"bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1\": container with ID starting with bc91b98331fe0ebb1f666ca24aea753e29d6e1d8b01bcaf8fdbcf1fa5f1a22b1 not found: ID does not exist" Nov 01 22:56:01.704691 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.704647 5778 scope.go:122] "RemoveContainer" containerID="75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a" Nov 01 22:56:01.714398 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.714374 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:56:01.718796 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.718775 5778 scope.go:122] "RemoveContainer" containerID="75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a" Nov 01 22:56:01.719186 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:01.719160 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a\": container with ID starting with 75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a not found: ID does not exist" containerID="75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a" Nov 01 22:56:01.719274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.719208 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a"} err="failed to get container status \"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a\": rpc error: code = NotFound desc = could not find container \"75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a\": container with ID starting with 75eb8e8974ade4c09cf7cf9e602f318fc1a70c4231903554a0b3a8866038df0a not found: ID does not exist" Nov 01 22:56:01.719274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.719233 5778 scope.go:122] "RemoveContainer" containerID="bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867" Nov 01 22:56:01.720339 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.720254 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khd85"] Nov 01 22:56:01.728967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.728946 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:56:01.738374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.738350 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-c4z56"] Nov 01 22:56:01.740270 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.740241 5778 scope.go:122] "RemoveContainer" containerID="bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867" Nov 01 22:56:01.740580 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:01.740558 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867\": container with ID starting with bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867 not found: ID does not exist" containerID="bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867" Nov 01 22:56:01.740650 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.740587 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867"} err="failed to get container status \"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867\": rpc error: code = NotFound desc = could not find container \"bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867\": container with ID starting with bb3e661de768897a6fa782fe904f4795c6f5a7df245182c777f11c40773db867 not found: ID does not exist" Nov 01 22:56:01.752693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.752670 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:56:01.758717 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:01.758695 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-mv6nf"] Nov 01 22:56:02.376263 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:02.376231 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" path="/var/lib/kubelet/pods/5369d805-44aa-4cc2-9c18-b93525ab9b61/volumes" Nov 01 22:56:02.376764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:02.376750 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" path="/var/lib/kubelet/pods/565eebb6-d87e-4f1c-bae0-e78d741850d4/volumes" Nov 01 22:56:02.377226 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:02.377212 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" path="/var/lib/kubelet/pods/5b6ebf20-b0bd-4afe-8a67-de7f357db9f8/volumes" Nov 01 22:56:02.378004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:02.377988 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" path="/var/lib/kubelet/pods/5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4/volumes" Nov 01 22:56:07.574537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:07.573665 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:56:07.574537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:07.573955 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" podUID="0f58e093-c761-4238-9205-39f663f29985" containerName="maas-api" containerID="cri-o://290db7ed06e29d6e41590d17771fc358bd38cfb014d1bc377287350bca09ff8c" gracePeriod=30 Nov 01 22:56:07.858679 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:07.858279 5778 generic.go:356] "Generic (PLEG): container finished" podID="0f58e093-c761-4238-9205-39f663f29985" containerID="290db7ed06e29d6e41590d17771fc358bd38cfb014d1bc377287350bca09ff8c" exitCode=0 Nov 01 22:56:07.858679 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:07.858531 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" event={"ID":"0f58e093-c761-4238-9205-39f663f29985","Type":"ContainerDied","Data":"290db7ed06e29d6e41590d17771fc358bd38cfb014d1bc377287350bca09ff8c"} Nov 01 22:56:08.548455 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.548426 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:56:08.662906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.662871 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") pod \"0f58e093-c761-4238-9205-39f663f29985\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " Nov 01 22:56:08.663361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.663071 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\") pod \"0f58e093-c761-4238-9205-39f663f29985\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " Nov 01 22:56:08.663361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.663205 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\" (UniqueName: \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\") pod \"0f58e093-c761-4238-9205-39f663f29985\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " Nov 01 22:56:08.663361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.663307 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\") pod \"0f58e093-c761-4238-9205-39f663f29985\" (UID: \"0f58e093-c761-4238-9205-39f663f29985\") " Nov 01 22:56:08.663552 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.663509 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca" pod "0f58e093-c761-4238-9205-39f663f29985" (UID: "0f58e093-c761-4238-9205-39f663f29985"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:08.664113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.664070 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58e093-c761-4238-9205-39f663f29985-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:08.665684 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.665657 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns" pod "0f58e093-c761-4238-9205-39f663f29985" (UID: "0f58e093-c761-4238-9205-39f663f29985"). InnerVolumeSpecName "kube-api-access-mlnns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:08.665780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.665684 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs" pod "0f58e093-c761-4238-9205-39f663f29985" (UID: "0f58e093-c761-4238-9205-39f663f29985"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:08.665824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.665795 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls" pod "0f58e093-c761-4238-9205-39f663f29985" (UID: "0f58e093-c761-4238-9205-39f663f29985"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:08.765550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.765510 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnns\" (UniqueName: \"kubernetes.io/projected/0f58e093-c761-4238-9205-39f663f29985-kube-api-access-mlnns\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:08.765550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.765548 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:08.765771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.765562 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58e093-c761-4238-9205-39f663f29985-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:08.868317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.868283 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" Nov 01 22:56:08.868317 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.868297 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt" event={"ID":"0f58e093-c761-4238-9205-39f663f29985","Type":"ContainerDied","Data":"55dee19c7fe41e61d9cdf8fd811fc8dab0549c113b4f654872b2ff1285e83846"} Nov 01 22:56:08.868554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.868379 5778 scope.go:122] "RemoveContainer" containerID="290db7ed06e29d6e41590d17771fc358bd38cfb014d1bc377287350bca09ff8c" Nov 01 22:56:08.894008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.893979 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:56:08.898550 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:08.898522 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58c957-79f7b6f6d9-9phdt"] Nov 01 22:56:10.377130 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:10.377099 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f58e093-c761-4238-9205-39f663f29985" path="/var/lib/kubelet/pods/0f58e093-c761-4238-9205-39f663f29985/volumes" Nov 01 22:56:12.329878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.325932 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.420875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.420079 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.420875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.420453 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.480876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.480073 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.144:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.572876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.571994 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.150:8000/ready\": context deadline exceeded" Nov 01 22:56:12.594938 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.580717 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.598875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.596473 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.694878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.673300 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.820647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.814051 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.853395 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.853289 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 01 22:56:12.901558 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:12.901508 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.063039 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.062983 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.076109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.076057 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" probeResult="failure" output="Get \"https://10.132.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.179070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.179009 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 01 22:56:13.180374 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.180327 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.654877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.646871 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": context deadline exceeded" Nov 01 22:56:13.686919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.685692 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.686919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.686136 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:56:13.692218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.692172 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": context deadline exceeded" Nov 01 22:56:13.787935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.702757 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:56:13.970878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.873279 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.970878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:13.873781 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:56:14.072933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.067067 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:56:14.072933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.068433 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:56:14.111587 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.111557 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 22:56:14.122783 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.122436 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 22:56:14.217877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.213399 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 22:56:14.411482 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:14.398619 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 22:56:17.162564 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:17.162522 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:56:17.163148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:17.162825 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" containerID="cri-o://5ad7205a852bd3d848fd28d1ec047707e002d905b9dc88e16b1a3e3d29dbca34" gracePeriod=30 Nov 01 22:56:17.297996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:17.285059 5778 patch_prober.go:29] interesting pod/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:56:17.297996 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:17.285142 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:56:18.777326 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:18.777287 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:56:22.265049 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.265016 5778 generic.go:356] "Generic (PLEG): container finished" podID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerID="5ad7205a852bd3d848fd28d1ec047707e002d905b9dc88e16b1a3e3d29dbca34" exitCode=0 Nov 01 22:56:22.265497 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.265048 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerDied","Data":"5ad7205a852bd3d848fd28d1ec047707e002d905b9dc88e16b1a3e3d29dbca34"} Nov 01 22:56:22.265497 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.265127 5778 scope.go:122] "RemoveContainer" containerID="3d123c8a102788ae4baba2b83a87d4def732ee523007dbd5de0e3d0a6920c7b9" Nov 01 22:56:22.957500 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.957438 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:56:22.970841 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.970813 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.970898 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971010 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.970928 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971127 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971087 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:22.971179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971146 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971234 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971210 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971243 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971258 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:22.971293 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971288 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971328 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971445 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971376 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\") pod \"7a2e10f5-e70d-493a-8004-ea98b990012f\" (UID: \"7a2e10f5-e70d-493a-8004-ea98b990012f\") " Nov 01 22:56:22.971799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971650 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:22.971947 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971751 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:22.972024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.971786 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:22.972245 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.972228 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2e10f5-e70d-493a-8004-ea98b990012f-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.972340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.972248 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.972340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.972262 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.972340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.972276 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.972340 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.972291 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.973478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.973423 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:22.973789 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.973771 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:22.974085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.974063 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:22.974085 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:22.974073 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6" pod "7a2e10f5-e70d-493a-8004-ea98b990012f" (UID: "7a2e10f5-e70d-493a-8004-ea98b990012f"). InnerVolumeSpecName "kube-api-access-q5cw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:23.076635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.076601 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:23.076635 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.076635 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:23.076962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.076649 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cw6\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-kube-api-access-q5cw6\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:23.076962 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.076677 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2e10f5-e70d-493a-8004-ea98b990012f-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:23.266928 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.266663 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:56:23.267409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.266985 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" containerID="cri-o://c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3" gracePeriod=30 Nov 01 22:56:23.291312 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.291278 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" Nov 01 22:56:23.291480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.291408 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f" event={"ID":"7a2e10f5-e70d-493a-8004-ea98b990012f","Type":"ContainerDied","Data":"ef976b098154c4cb45122b7e4da86cac1ecf5532dc2416b18fc5a8972246840b"} Nov 01 22:56:23.291480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.291465 5778 scope.go:122] "RemoveContainer" containerID="5ad7205a852bd3d848fd28d1ec047707e002d905b9dc88e16b1a3e3d29dbca34" Nov 01 22:56:23.341876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.340351 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:56:23.423009 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:23.418990 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-58c957-openshift-default-58b6557565-nv77f"] Nov 01 22:56:24.066205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.066163 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 22:56:24.067393 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067364 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067395 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067413 5778 cpu_manager.go:414] "containerMap: removing container" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067423 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067440 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067449 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067459 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067467 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067480 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:24.067522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067487 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067536 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0f58e093-c761-4238-9205-39f663f29985" containerName="maas-api" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067544 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f58e093-c761-4238-9205-39f663f29985" containerName="maas-api" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067575 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067583 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067600 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067607 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067623 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067630 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067822 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067838 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9b8603-13ce-4f85-9737-83efaeed82a2" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067866 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee5b60b-2e8b-47b9-9ef4-d03c45cc3fa5" containerName="main" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067882 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5369d805-44aa-4cc2-9c18-b93525ab9b61" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067892 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="565eebb6-d87e-4f1c-bae0-e78d741850d4" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067903 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6ebf20-b0bd-4afe-8a67-de7f357db9f8" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067914 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e70486d-59c4-4b4c-bcb3-317e7f1e7cb4" containerName="istio-proxy" Nov 01 22:56:24.067933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.067932 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f58e093-c761-4238-9205-39f663f29985" containerName="maas-api" Nov 01 22:56:24.088644 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.088608 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.090694 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.090476 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 22:56:24.093579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.093555 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-53dd05-openshift-default-dockercfg-4ppx6\"" Nov 01 22:56:24.162218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.162156 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.162705 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.162684 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.163014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.162995 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.163214 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.163181 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.163406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.163357 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.163602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.163554 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.163813 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.163798 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kngl\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.164059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.164033 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.164345 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.164329 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266018 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kngl\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266097 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266230 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266317 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266449 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266522 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266558 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266589 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.266954 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.266648 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.271628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.271594 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.272146 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.272119 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.272660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.272638 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.273467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.273445 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.273982 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.273961 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.287303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.287267 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.400047 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.399927 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.400893 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.400830 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kngl\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.415097 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.415069 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.422328 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.422306 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:24.455260 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:24.454694 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" path="/var/lib/kubelet/pods/7a2e10f5-e70d-493a-8004-ea98b990012f/volumes" Nov 01 22:56:25.062531 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.062498 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 22:56:25.062834 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:56:25.062808 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea46be7d_a060_4d72_91db_e4aabfbd2bec.slice/crio-d02804ebbb69b5576383dfff54d065e1c9389225c7d61825dba00c52a9a1888c WatchSource:0}: Error finding container d02804ebbb69b5576383dfff54d065e1c9389225c7d61825dba00c52a9a1888c: Status 404 returned error can't find the container with id d02804ebbb69b5576383dfff54d065e1c9389225c7d61825dba00c52a9a1888c Nov 01 22:56:25.079961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.079655 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:25.080080 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.080028 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:25.080122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.080079 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:25.384920 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.384821 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerStarted","Data":"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27"} Nov 01 22:56:25.385321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.384937 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerStarted","Data":"d02804ebbb69b5576383dfff54d065e1c9389225c7d61825dba00c52a9a1888c"} Nov 01 22:56:25.408824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.408778 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podStartSLOduration=1.408764285 podStartE2EDuration="1.408764285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:25.406705056 +0000 UTC m=+1955.182691886" watchObservedRunningTime="2026-11-01 22:56:25.408764285 +0000 UTC m=+1955.184751114" Nov 01 22:56:25.432628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:25.432592 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:26.437202 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:26.437170 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:27.402924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:27.402874 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:27.403754 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:27.403730 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:56:29.001706 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.001668 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:56:29.085766 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.085735 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.085991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.085777 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.085991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.085818 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.085991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.085927 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.085991 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.085957 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.086225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086011 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.086225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086041 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.086225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086082 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.086225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086089 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.086225 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086153 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.086478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086244 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.086478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086320 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.086478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086375 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\") pod \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\" (UID: \"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845\") " Nov 01 22:56:29.086935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086919 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.087005 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086937 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.087005 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086948 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.087005 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.086956 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.087222 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.087208 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:29.088686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.088657 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.089785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.089753 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "kube-api-access-bvgvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.090542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.090515 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.090673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.090652 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo" pod "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" (UID: "b0c3560c-cbf5-4e5a-bb54-9224c4fe1845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:29.188576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.188517 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.188576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.188558 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.188576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.188573 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.189078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.188587 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.189078 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.188601 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgvs\" (UniqueName: \"kubernetes.io/projected/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845-kube-api-access-bvgvs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.610103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.606414 5778 generic.go:356] "Generic (PLEG): container finished" podID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerID="c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3" exitCode=0 Nov 01 22:56:29.610103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.607344 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" Nov 01 22:56:29.610103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.607794 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" event={"ID":"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845","Type":"ContainerDied","Data":"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3"} Nov 01 22:56:29.610103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.607866 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm" event={"ID":"b0c3560c-cbf5-4e5a-bb54-9224c4fe1845","Type":"ContainerDied","Data":"2e31089024285cfbe407a53ce22ab2d1947c3e8742c2838e440d1d7c1a30b182"} Nov 01 22:56:29.610103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.607901 5778 scope.go:122] "RemoveContainer" containerID="c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3" Nov 01 22:56:29.628566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.628499 5778 scope.go:122] "RemoveContainer" containerID="c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3" Nov 01 22:56:29.628819 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:29.628792 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3\": container with ID starting with c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3 not found: ID does not exist" containerID="c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3" Nov 01 22:56:29.628951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.628827 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3"} err="failed to get container status \"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3\": rpc error: code = NotFound desc = could not find container \"c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3\": container with ID starting with c0f8b88da0a63f0e84c27ac072df3e2f26edba643cd10d58ce6dc160c7eaa6a3 not found: ID does not exist" Nov 01 22:56:29.644576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.644549 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:56:29.651595 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:29.651568 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-wvpcm"] Nov 01 22:56:30.376442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:30.376410 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" path="/var/lib/kubelet/pods/b0c3560c-cbf5-4e5a-bb54-9224c4fe1845/volumes" Nov 01 22:56:35.927910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.927872 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 22:56:35.928887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.928865 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" Nov 01 22:56:35.928887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.928887 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" Nov 01 22:56:35.929084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.929061 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c3560c-cbf5-4e5a-bb54-9224c4fe1845" containerName="istio-proxy" Nov 01 22:56:35.929084 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.929082 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e10f5-e70d-493a-8004-ea98b990012f" containerName="istio-proxy" Nov 01 22:56:35.934513 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.934490 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.942167 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.942143 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-64b196-openshift-default-dockercfg-h5sfj\"" Nov 01 22:56:35.947802 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.947745 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 22:56:35.986655 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.986621 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qt9\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.986793 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.986775 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.986988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.986970 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987014 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987182 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987165 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987204 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987252 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987229 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987396 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987372 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:35.987454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:35.987419 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.088999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.088965 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.088999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089006 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089101 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089130 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089157 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089196 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089226 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089348 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qt9\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089439 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.089758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.089659 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.090213 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.090169 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.090213 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.090193 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.090386 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.090311 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.090630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.090611 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.092686 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.092661 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.093041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.093020 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.098205 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.098184 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.098593 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.098564 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qt9\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.269674 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.269597 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:36.591026 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.590996 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 22:56:36.593584 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:56:36.593553 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb327de33_07f0_4316_8a69_89e0211836f3.slice/crio-68c54de8ba11b187f152cf9424e2ded75be27b3bda4dd004664b650ade936b0a WatchSource:0}: Error finding container 68c54de8ba11b187f152cf9424e2ded75be27b3bda4dd004664b650ade936b0a: Status 404 returned error can't find the container with id 68c54de8ba11b187f152cf9424e2ded75be27b3bda4dd004664b650ade936b0a Nov 01 22:56:36.597189 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.597150 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:36.597292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.597235 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:36.597292 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.597284 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:36.684169 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.684142 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" event={"ID":"b327de33-07f0-4316-8a69-89e0211836f3","Type":"ContainerStarted","Data":"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765"} Nov 01 22:56:36.684274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.684197 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" event={"ID":"b327de33-07f0-4316-8a69-89e0211836f3","Type":"ContainerStarted","Data":"68c54de8ba11b187f152cf9424e2ded75be27b3bda4dd004664b650ade936b0a"} Nov 01 22:56:36.713056 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:36.712997 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podStartSLOduration=1.712975323 podStartE2EDuration="1.712975323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:36.708795738 +0000 UTC m=+1966.484782566" watchObservedRunningTime="2026-11-01 22:56:36.712975323 +0000 UTC m=+1966.488962154" Nov 01 22:56:37.272501 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:37.272455 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:37.273195 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:37.273000 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:37.273282 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:37.273241 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:38.270783 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:38.270743 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:38.271001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:38.270840 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:39.270303 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.270270 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:39.270887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.270448 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:39.764819 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.764786 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:56:39.769783 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.769762 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.773579 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.773552 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e685d-openshift-default-dockercfg-w4dhv\"" Nov 01 22:56:39.787791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.787763 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797089 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797148 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797303 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797340 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797402 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797468 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797518 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28mg\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797609 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.799239 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.797647 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.898881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.898823 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.898915 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.898978 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899043 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899185 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899097 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28mg\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899221 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899187 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899226 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899356 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899339 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.899397 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899381 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.900065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899628 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.900065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899755 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.900065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.899764 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.900065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.900048 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.900322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.900073 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.902595 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.901795 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.903623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.903416 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.909014 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.908990 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:39.909228 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:39.909205 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28mg\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:40.090218 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.090139 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:40.270174 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.270135 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:40.270343 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.270201 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:40.496963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.493941 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:56:40.535911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.527527 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:40.535911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.527629 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:40.535911 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.527679 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:40.768806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.768725 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" event={"ID":"459bfc6b-e188-4475-9516-926afc86f1d2","Type":"ContainerStarted","Data":"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0"} Nov 01 22:56:40.768806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:40.768790 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" event={"ID":"459bfc6b-e188-4475-9516-926afc86f1d2","Type":"ContainerStarted","Data":"8d1e0da18868d0b128998be5b01f7ea071f998cd741d0fa1dfb537e8dab2a66a"} Nov 01 22:56:41.090904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:41.090801 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:41.092400 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:41.092374 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:41.092529 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:41.092426 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:41.288885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:41.281879 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:41.288885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:41.281937 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:42.107882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:42.107082 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:42.107882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:42.107149 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:42.272880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:42.270320 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:56:42.272880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:42.270410 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 22:56:43.091122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.091092 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:43.091624 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.091187 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:43.491625 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.491592 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:43.493100 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.493072 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:43.495206 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.495182 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 22:56:43.516200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:43.516156 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podStartSLOduration=4.516143275 podStartE2EDuration="4.516143275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:40.796144763 +0000 UTC m=+1970.572131597" watchObservedRunningTime="2026-11-01 22:56:43.516143275 +0000 UTC m=+1973.292130104" Nov 01 22:56:44.091485 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:44.091351 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:44.091485 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:44.091408 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:45.090561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.090522 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:45.090728 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.090585 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:45.302963 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.302924 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 22:56:45.308154 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.308131 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.357666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.357595 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 22:56:45.395353 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.395320 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f675d-openshift-default-dockercfg-hv7v4\"" Nov 01 22:56:45.470481 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470445 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470514 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470607 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470627 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470664 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470824 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjr2f\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.470999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.470979 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.471057 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.471037 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.471115 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.471097 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572348 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572426 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572500 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572576 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572551 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572756 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572583 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572828 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjr2f\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572953 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.572976 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.572967 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573050 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573101 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573106 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573180 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573370 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573351 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.573453 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.573423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.575383 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.575360 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.575537 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.575515 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.581787 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.581762 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.581923 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.581793 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjr2f\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.655431 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.655401 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:45.999256 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:56:45.999176 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ad79e_fbca_4caf_a90c_e7c832ff4647.slice/crio-24154714ecebbaa756eb90b217b305772f4bba25d240590d8520befd5cc0401d WatchSource:0}: Error finding container 24154714ecebbaa756eb90b217b305772f4bba25d240590d8520befd5cc0401d: Status 404 returned error can't find the container with id 24154714ecebbaa756eb90b217b305772f4bba25d240590d8520befd5cc0401d Nov 01 22:56:45.999480 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:45.999442 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 22:56:46.002532 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.002500 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:46.002611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.002567 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:46.002611 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.002595 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:56:46.090910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.090877 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:46.091019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.090931 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:46.990002 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.989966 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" event={"ID":"3d9ad79e-fbca-4caf-a90c-e7c832ff4647","Type":"ContainerStarted","Data":"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73"} Nov 01 22:56:46.990404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:46.990018 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" event={"ID":"3d9ad79e-fbca-4caf-a90c-e7c832ff4647","Type":"ContainerStarted","Data":"24154714ecebbaa756eb90b217b305772f4bba25d240590d8520befd5cc0401d"} Nov 01 22:56:47.013406 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.013357 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podStartSLOduration=2.013343548 podStartE2EDuration="2.013343548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:47.009076107 +0000 UTC m=+1976.785062936" watchObservedRunningTime="2026-11-01 22:56:47.013343548 +0000 UTC m=+1976.789330376" Nov 01 22:56:47.091195 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.091158 5778 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:56:47.091387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.091218 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 01 22:56:47.656238 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.656191 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:47.658425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.658389 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:47.658607 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:47.658451 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:48.175875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:48.161155 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:48.175875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:48.162571 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:48.175875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:48.164058 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:56:48.656077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:48.656026 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:48.656264 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:48.656103 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:48.777619 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:48.777572 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:56:49.108359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:49.108279 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:56:49.108539 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:49.108458 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" containerID="cri-o://f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a" gracePeriod=30 Nov 01 22:56:49.656700 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:49.656667 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:49.657663 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:49.657625 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:50.663886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:50.662129 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:50.663886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:50.662195 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:51.289732 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:51.289681 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:56:51.289999 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:51.289963 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" containerID="cri-o://45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83" gracePeriod=30 Nov 01 22:56:51.662389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:51.661873 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:51.662389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:51.661966 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:52.586107 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.586075 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 22:56:52.597747 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.597719 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 22:56:52.597905 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.597808 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.602563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.602537 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c6cec6\"/\"kube-root-ca.crt\"" Nov 01 22:56:52.602685 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.602597 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c6cec6\"/\"openshift-service-ca.crt\"" Nov 01 22:56:52.602685 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.602546 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c6cec6\"/\"default-dockercfg-mmxjv\"" Nov 01 22:56:52.602685 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.602621 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c6cec6\"/\"auth-test-model-c6cec6-kserve-self-signed-certs\"" Nov 01 22:56:52.675758 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.675725 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:52.676092 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.676064 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736335 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736507 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736765 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736820 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736872 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928sf\" (UniqueName: \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.742627 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.736914 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.839929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.839349 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928sf\" (UniqueName: \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.839929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.839429 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.839929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.839582 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.839929 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.839742 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.843623 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.840836 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.850554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.850522 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.850702 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.850593 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.862902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.851034 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.862902 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.851465 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.876874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.867285 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-928sf\" (UniqueName: \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.878465 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.878439 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.903295 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.903227 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\") pod \"auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:52.916630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:52.916351 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:53.584282 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:53.584200 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 22:56:53.589911 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:56:53.589877 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3829ba6_8666_41c2_8028_f1d132af429d.slice/crio-9bb85c12be01b710b298ed329a42399dacc3e5c769ce15292bbd1e2285a3dcad WatchSource:0}: Error finding container 9bb85c12be01b710b298ed329a42399dacc3e5c769ce15292bbd1e2285a3dcad: Status 404 returned error can't find the container with id 9bb85c12be01b710b298ed329a42399dacc3e5c769ce15292bbd1e2285a3dcad Nov 01 22:56:53.662448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:53.662340 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:53.662448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:53.662393 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 22:56:54.285891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.283721 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" event={"ID":"d3829ba6-8666-41c2-8028-f1d132af429d","Type":"ContainerStarted","Data":"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42"} Nov 01 22:56:54.285891 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.283778 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" event={"ID":"d3829ba6-8666-41c2-8028-f1d132af429d","Type":"ContainerStarted","Data":"9bb85c12be01b710b298ed329a42399dacc3e5c769ce15292bbd1e2285a3dcad"} Nov 01 22:56:54.347888 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.344492 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" podStartSLOduration=2.34447165 podStartE2EDuration="2.34447165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:54.316438783 +0000 UTC m=+1984.092425615" watchObservedRunningTime="2026-11-01 22:56:54.34447165 +0000 UTC m=+1984.120458482" Nov 01 22:56:54.737874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.737153 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:54.761901 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.740154 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:54.848638 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.846208 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 22:56:54.854528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.853396 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-48tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 01 22:56:54.854528 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.853454 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 01 22:56:54.919606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:54.919128 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:56:55.120824 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.120792 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:56:55.206423 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206390 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206432 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206483 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206608 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206536 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206662 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206801 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206744 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206901 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.206975 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.206933 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.207087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207038 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\") pod \"3f46718b-8c9e-4d01-9486-e1a76d295976\" (UID: \"3f46718b-8c9e-4d01-9486-e1a76d295976\") " Nov 01 22:56:55.207148 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207073 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:55.207335 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207311 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:55.207565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207547 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.207631 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207567 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.207692 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207624 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:55.208286 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.207903 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:55.208286 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.208149 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:55.210973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.210579 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "kube-api-access-b78kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:55.210973 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.210644 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:55.226567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.226538 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:55.228181 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.228159 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo" pod "3f46718b-8c9e-4d01-9486-e1a76d295976" (UID: "3f46718b-8c9e-4d01-9486-e1a76d295976"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:55.294162 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.294133 5778 generic.go:356] "Generic (PLEG): container finished" podID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerID="f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a" exitCode=0 Nov 01 22:56:55.294272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.294181 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" event={"ID":"3f46718b-8c9e-4d01-9486-e1a76d295976","Type":"ContainerDied","Data":"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a"} Nov 01 22:56:55.294272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.294239 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" event={"ID":"3f46718b-8c9e-4d01-9486-e1a76d295976","Type":"ContainerDied","Data":"b4c59b05a7c9016f7d6ecdd7a3af571f0f0d01c8a2fb3488703adf26a251a714"} Nov 01 22:56:55.294272 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.294267 5778 scope.go:122] "RemoveContainer" containerID="f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a" Nov 01 22:56:55.294448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.294190 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss" Nov 01 22:56:55.307524 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.307432 5778 scope.go:122] "RemoveContainer" containerID="f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a" Nov 01 22:56:55.307823 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:55.307799 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a\": container with ID starting with f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a not found: ID does not exist" containerID="f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a" Nov 01 22:56:55.308156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.307834 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a"} err="failed to get container status \"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a\": rpc error: code = NotFound desc = could not find container \"f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a\": container with ID starting with f28926a28bf4481e67c6ea8acd8b96c9d3628bf137785366e83be2d571b3a70a not found: ID does not exist" Nov 01 22:56:55.309310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309285 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309316 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-kube-api-access-b78kp\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309333 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309348 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309364 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309375 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46718b-8c9e-4d01-9486-e1a76d295976-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.309399 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.309384 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46718b-8c9e-4d01-9486-e1a76d295976-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.340759 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.340670 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:56:55.347926 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:55.347890 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-48tss"] Nov 01 22:56:56.376830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.376801 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" path="/var/lib/kubelet/pods/3f46718b-8c9e-4d01-9486-e1a76d295976/volumes" Nov 01 22:56:56.714530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.714504 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:56:56.805955 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.805918 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806008 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806078 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806127 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806177 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806344 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806371 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806395 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806474 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806501 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806588 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\") pod \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\" (UID: \"5596ee8f-f355-4d12-bb50-aef28f0dbcf3\") " Nov 01 22:56:56.806159 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806626 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:56.807461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.806819 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:56.807461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807005 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:56.807628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807590 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.807628 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807620 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.807749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807636 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.807749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807649 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.807749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.807664 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.810164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.810140 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "kube-api-access-rlmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:56.810283 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.810256 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:56.810782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.810757 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:56.810940 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.810915 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo" pod "5596ee8f-f355-4d12-bb50-aef28f0dbcf3" (UID: "5596ee8f-f355-4d12-bb50-aef28f0dbcf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:56.909897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.909865 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.909897 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.909895 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmtk\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-kube-api-access-rlmtk\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.910061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.909906 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:56.910061 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:56.909920 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5596ee8f-f355-4d12-bb50-aef28f0dbcf3-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:57.317708 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.317668 5778 generic.go:356] "Generic (PLEG): container finished" podID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerID="45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83" exitCode=0 Nov 01 22:56:57.318054 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.318008 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerDied","Data":"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83"} Nov 01 22:56:57.318153 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.318068 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" event={"ID":"5596ee8f-f355-4d12-bb50-aef28f0dbcf3","Type":"ContainerDied","Data":"d074f488ddbfbd280003c6780d24599b7291bb9e394abecd723774a6f2afbdc5"} Nov 01 22:56:57.318153 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.318103 5778 scope.go:122] "RemoveContainer" containerID="45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83" Nov 01 22:56:57.318280 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.318256 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2" Nov 01 22:56:57.343460 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.343434 5778 scope.go:122] "RemoveContainer" containerID="e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf" Nov 01 22:56:57.354563 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.354538 5778 scope.go:122] "RemoveContainer" containerID="45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83" Nov 01 22:56:57.354978 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:57.354922 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83\": container with ID starting with 45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83 not found: ID does not exist" containerID="45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83" Nov 01 22:56:57.355059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.354968 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83"} err="failed to get container status \"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83\": rpc error: code = NotFound desc = could not find container \"45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83\": container with ID starting with 45a805e7c7105d928428dc1fe7d38b40a0c0c1236c4fabebb9ec6b4daaa91c83 not found: ID does not exist" Nov 01 22:56:57.355059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.354993 5778 scope.go:122] "RemoveContainer" containerID="e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf" Nov 01 22:56:57.355352 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:56:57.355330 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf\": container with ID starting with e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf not found: ID does not exist" containerID="e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf" Nov 01 22:56:57.355427 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.355361 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf"} err="failed to get container status \"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf\": rpc error: code = NotFound desc = could not find container \"e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf\": container with ID starting with e9cf16bc97e3dbf1b15327df86057bdbd5bb6d7337f974c1a1700d94cfc36ddf not found: ID does not exist" Nov 01 22:56:57.376773 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.376752 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:56:57.388534 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:57.388506 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e716d-openshift-default-857f6685b7-wxxg2"] Nov 01 22:56:58.376553 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:56:58.376516 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" path="/var/lib/kubelet/pods/5596ee8f-f355-4d12-bb50-aef28f0dbcf3/volumes" Nov 01 22:57:02.917835 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:02.917792 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:57:02.937122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:02.937088 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:57:03.725004 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:03.723283 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 22:57:13.765121 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.765023 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 22:57:13.766865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766816 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.766992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766878 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.766992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766906 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.766992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766915 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.766992 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766987 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" Nov 01 22:57:13.767255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.766998 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" Nov 01 22:57:13.767255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.767240 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f46718b-8c9e-4d01-9486-e1a76d295976" containerName="istio-proxy" Nov 01 22:57:13.767336 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.767294 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.767336 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.767310 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5596ee8f-f355-4d12-bb50-aef28f0dbcf3" containerName="istio-proxy" Nov 01 22:57:13.773656 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.773631 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 22:57:13.773809 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.773736 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.784095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.784071 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-947846\"/\"kube-root-ca.crt\"" Nov 01 22:57:13.784248 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.784158 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-947846\"/\"openshift-service-ca.crt\"" Nov 01 22:57:13.785255 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.785227 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-947846\"/\"default-dockercfg-vmkh6\"" Nov 01 22:57:13.785487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.785465 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-947846\"/\"auth-test-model-947846-kserve-self-signed-certs\"" Nov 01 22:57:13.894921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.894880 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.895120 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.895036 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.895200 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.895183 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.895322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.895299 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.895478 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.895456 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2skx\" (UniqueName: \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:13.895533 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:13.895520 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.001691 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2skx\" (UniqueName: \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.001769 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.001939 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.002000 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.002102 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.002164 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.002649 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.004879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.003926 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.017921 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.014700 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.047871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.037805 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.047871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.040563 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2skx\" (UniqueName: \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.052878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.050260 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\") pod \"auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.116885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.111984 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:14.660179 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.651330 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 22:57:14.916249 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:14.916219 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" event={"ID":"56f8d433-c175-4e32-8c33-74f4cbd1603c","Type":"ContainerStarted","Data":"c3d1560f59214263df9f53b5b98380659fe6ac9ba6dd9941667c08c911df610a"} Nov 01 22:57:15.928773 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:15.928735 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" event={"ID":"56f8d433-c175-4e32-8c33-74f4cbd1603c","Type":"ContainerStarted","Data":"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca"} Nov 01 22:57:15.949094 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:15.949040 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" podStartSLOduration=2.94902344 podStartE2EDuration="2.94902344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:15.947080868 +0000 UTC m=+2005.723067720" watchObservedRunningTime="2026-11-01 22:57:15.94902344 +0000 UTC m=+2005.725010268" Nov 01 22:57:16.113865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:16.113813 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:18.776349 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:57:18.776297 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:57:24.112595 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:24.112559 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:24.125494 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:24.125468 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:25.026193 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:25.026161 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 22:57:28.975761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.975726 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 22:57:28.982133 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.982107 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:28.990542 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.990515 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 22:57:28.990949 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.990926 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-49ab43\"/\"default-dockercfg-cmdsv\"" Nov 01 22:57:28.991164 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.991140 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-49ab43\"/\"auth-test-model-49ab43-kserve-self-signed-certs\"" Nov 01 22:57:28.991278 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.991204 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-49ab43\"/\"openshift-service-ca.crt\"" Nov 01 22:57:28.991402 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:28.991312 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-49ab43\"/\"kube-root-ca.crt\"" Nov 01 22:57:29.117450 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117394 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg44r\" (UniqueName: \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.117667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117462 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.117667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117510 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.117667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117527 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.117667 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117579 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.117919 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.117732 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.222003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.221960 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg44r\" (UniqueName: \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.222606 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.222580 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.222806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.222791 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.238409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.237978 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.238699 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.238678 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.239433 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.239405 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.247262 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.234341 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.247441 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.239871 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.247541 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.238332 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.258870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.249689 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg44r\" (UniqueName: \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.258870 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.256151 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.276871 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.268786 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\") pod \"auth-test-model-49ab43-kserve-86d57c8df7-m8xlw\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.305507 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.305172 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:29.744930 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:57:29.744894 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd116b84_4775_47c3_8981_c1c6088a6289.slice/crio-ef193b207843309694de5238154897f5d812634c3d627f82d04dd8e1bd5b69bf WatchSource:0}: Error finding container ef193b207843309694de5238154897f5d812634c3d627f82d04dd8e1bd5b69bf: Status 404 returned error can't find the container with id ef193b207843309694de5238154897f5d812634c3d627f82d04dd8e1bd5b69bf Nov 01 22:57:29.746459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:29.746431 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 22:57:30.076304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:30.076213 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" event={"ID":"cd116b84-4775-47c3-8981-c1c6088a6289","Type":"ContainerStarted","Data":"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614"} Nov 01 22:57:30.076304 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:30.076270 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" event={"ID":"cd116b84-4775-47c3-8981-c1c6088a6289","Type":"ContainerStarted","Data":"ef193b207843309694de5238154897f5d812634c3d627f82d04dd8e1bd5b69bf"} Nov 01 22:57:30.098457 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:30.098410 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" podStartSLOduration=2.098396161 podStartE2EDuration="2.098396161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:30.096819117 +0000 UTC m=+2019.872805960" watchObservedRunningTime="2026-11-01 22:57:30.098396161 +0000 UTC m=+2019.874382990" Nov 01 22:57:30.307040 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:30.306972 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:39.306376 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:39.306335 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:39.320282 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:39.320250 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:40.215791 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:40.215757 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 22:57:43.928031 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:43.926225 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:57:48.799603 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:57:48.799566 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:57:59.009775 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.009708 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" containerID="cri-o://8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b" gracePeriod=45 Nov 01 22:57:59.313098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.313065 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 22:57:59.377288 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.377253 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:57:59.382687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.382656 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.390660 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.390609 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-53dd05\"/\"openshift-service-ca.crt\"" Nov 01 22:57:59.390806 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.390710 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-53dd05\"/\"kube-root-ca.crt\"" Nov 01 22:57:59.394375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.394348 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-53dd05\"/\"auth-test-model-53dd05-kserve-self-signed-certs\"" Nov 01 22:57:59.394561 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.394536 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 22:57:59.394651 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.394542 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-53dd05\"/\"default-dockercfg-qt9pj\"" Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586199 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586322 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586452 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586553 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\" (UniqueName: \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586648 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.586932 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.586693 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\") pod \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\" (UID: \"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35\") " Nov 01 22:57:59.587394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587061 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.587394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587179 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fr2\" (UniqueName: \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.587394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587255 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.587394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587332 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.587394 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587382 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.587643 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.587535 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.678983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.590151 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:59.678983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.650159 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:59.678983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.652221 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:59.678983 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.654968 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:57:59.723878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.717655 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.723878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.717791 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fr2\" (UniqueName: \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.723878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.717888 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.724425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.724390 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:59.733035 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.732576 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.742944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.736995 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fr2\" (UniqueName: \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.742944 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.737312 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.766886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.743494 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.776101 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.717963 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.776462 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.776363 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.776825 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.776729 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.783330 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.783293 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:59.783714 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.783646 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.783941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.783611 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:59.783941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.783875 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:59.783941 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.783891 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:59.784389 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.784276 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:59.786799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.786778 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\") pod \"auth-test-model-53dd05-kserve-59f8494757-wtlnd\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:57:59.789654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.788899 5778 generic.go:356] "Generic (PLEG): container finished" podID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerID="8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b" exitCode=0 Nov 01 22:57:59.789654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.789018 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" event={"ID":"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35","Type":"ContainerDied","Data":"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b"} Nov 01 22:57:59.789654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.789052 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" event={"ID":"9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35","Type":"ContainerDied","Data":"a3f61b702aa2d0abb0e39a570aa8a1db0001d2c4cbf152d0a4bab92ae1a77c62"} Nov 01 22:57:59.789654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.789073 5778 scope.go:122] "RemoveContainer" containerID="8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b" Nov 01 22:57:59.789654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.789260 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" Nov 01 22:57:59.795493 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.795455 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8" pod "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" (UID: "9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35"). InnerVolumeSpecName "kube-api-access-qhmn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:59.807671 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.807649 5778 scope.go:122] "RemoveContainer" containerID="8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b" Nov 01 22:57:59.808013 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:57:59.807991 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b\": container with ID starting with 8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b not found: ID does not exist" containerID="8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b" Nov 01 22:57:59.808117 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.808020 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b"} err="failed to get container status \"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b\": rpc error: code = NotFound desc = could not find container \"8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b\": container with ID starting with 8590f82bd953d475cfb71e99e300cca7a516dfe1852611f7f72dbe536321264b not found: ID does not exist" Nov 01 22:57:59.885699 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:57:59.885617 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmn8\" (UniqueName: \"kubernetes.io/projected/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35-kube-api-access-qhmn8\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:00.002630 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:00.002594 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:58:00.407959 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:00.392969 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:58:00.604294 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:00.591300 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:58:00.617935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:00.617901 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6b78bd/test-model-6b78bd-kserve-5cbf665d9-mj4qh"] Nov 01 22:58:00.815509 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:00.813080 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 22:58:00.823197 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:58:00.823162 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06e25a9_0262_46e4_8305_78dbd3f67389.slice/crio-db65559e0fe0d6575911ecd1911b416e751f0429ab27204416a571e2a1ced0aa WatchSource:0}: Error finding container db65559e0fe0d6575911ecd1911b416e751f0429ab27204416a571e2a1ced0aa: Status 404 returned error can't find the container with id db65559e0fe0d6575911ecd1911b416e751f0429ab27204416a571e2a1ced0aa Nov 01 22:58:01.370885 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:01.370108 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:58:01.832785 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:01.832743 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" event={"ID":"f06e25a9-0262-46e4-8305-78dbd3f67389","Type":"ContainerStarted","Data":"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb"} Nov 01 22:58:01.833325 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:01.832798 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" event={"ID":"f06e25a9-0262-46e4-8305-78dbd3f67389","Type":"ContainerStarted","Data":"db65559e0fe0d6575911ecd1911b416e751f0429ab27204416a571e2a1ced0aa"} Nov 01 22:58:01.853647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:01.853589 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" podStartSLOduration=2.853571553 podStartE2EDuration="2.853571553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:01.851176396 +0000 UTC m=+2051.627163249" watchObservedRunningTime="2026-11-01 22:58:01.853571553 +0000 UTC m=+2051.629558387" Nov 01 22:58:02.011882 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:02.007013 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:58:02.376647 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:02.376614 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" path="/var/lib/kubelet/pods/9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35/volumes" Nov 01 22:58:10.003784 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:10.003749 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:58:10.018024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:10.017998 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:58:10.932429 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:10.932401 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 22:58:16.396833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:16.396773 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" containerID="cri-o://b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de" gracePeriod=45 Nov 01 22:58:16.529876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:16.524358 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": dial tcp 10.132.0.134:8000: connect: connection refused" Nov 01 22:58:16.978961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:16.978925 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:58:17.065875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.065553 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.065875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.065636 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.065875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.065693 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.065875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.065760 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\" (UniqueName: \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.065875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.065834 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.066324 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.066008 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\") pod \"55a779fb-eaca-4e55-b27d-44d504bce1d3\" (UID: \"55a779fb-eaca-4e55-b27d-44d504bce1d3\") " Nov 01 22:58:17.076875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.071900 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:17.076875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.072115 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:17.076875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.073343 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:17.082654 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.082621 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "kube-api-access-k7fm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:17.082805 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.082769 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:17.084994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.084968 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs" pod "55a779fb-eaca-4e55-b27d-44d504bce1d3" (UID: "55a779fb-eaca-4e55-b27d-44d504bce1d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:58:17.171872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.171704 5778 generic.go:356] "Generic (PLEG): container finished" podID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerID="b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de" exitCode=0 Nov 01 22:58:17.172087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.171900 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" event={"ID":"55a779fb-eaca-4e55-b27d-44d504bce1d3","Type":"ContainerDied","Data":"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de"} Nov 01 22:58:17.172087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.171939 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" event={"ID":"55a779fb-eaca-4e55-b27d-44d504bce1d3","Type":"ContainerDied","Data":"d87c4b9019d78f6f0ee9b72c5d589c451d91be390c4bc3545d5474c93d83281b"} Nov 01 22:58:17.172087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.171960 5778 scope.go:122] "RemoveContainer" containerID="b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de" Nov 01 22:58:17.172240 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.172113 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177124 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177154 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177168 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177181 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a779fb-eaca-4e55-b27d-44d504bce1d3-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177196 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7fm8\" (UniqueName: \"kubernetes.io/projected/55a779fb-eaca-4e55-b27d-44d504bce1d3-kube-api-access-k7fm8\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.177879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.177210 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a779fb-eaca-4e55-b27d-44d504bce1d3-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:17.193074 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.193043 5778 scope.go:122] "RemoveContainer" containerID="b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de" Nov 01 22:58:17.193356 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:17.193332 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de\": container with ID starting with b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de not found: ID does not exist" containerID="b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de" Nov 01 22:58:17.193435 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.193367 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de"} err="failed to get container status \"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de\": rpc error: code = NotFound desc = could not find container \"b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de\": container with ID starting with b0dd0f1ef4ceebba99ab6da45a858e9f9c660a206fe38d59a10c56c4e18524de not found: ID does not exist" Nov 01 22:58:17.212896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.212135 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:58:17.237681 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:17.237637 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0e685d/test-model-0e685d-kserve-c6cbcf699-7gf5j"] Nov 01 22:58:18.430886 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:18.422695 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" path="/var/lib/kubelet/pods/55a779fb-eaca-4e55-b27d-44d504bce1d3/volumes" Nov 01 22:58:18.785315 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:18.785227 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:58:19.392022 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.391953 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 22:58:19.392790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392768 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" Nov 01 22:58:19.392790 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392787 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" Nov 01 22:58:19.392980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392804 5778 cpu_manager.go:414] "containerMap: removing container" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" Nov 01 22:58:19.392980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392810 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" Nov 01 22:58:19.392980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392961 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9caa97f7-2b0a-44b0-bc8d-3b32c82b5a35" containerName="main" Nov 01 22:58:19.392980 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.392969 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a779fb-eaca-4e55-b27d-44d504bce1d3" containerName="main" Nov 01 22:58:19.402689 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.400158 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.415522 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.415060 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-947846-openshift-default-dockercfg-qqfpg\"" Nov 01 22:58:19.419800 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.419459 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 22:58:19.476917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.476871 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477060 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477190 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477254 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477318 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477345 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477394 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslck\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477422 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.477830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.477478 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.578833 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.578800 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.578928 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.578974 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579037 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579067 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579054 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579102 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslck\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579128 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579182 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579332 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579348 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579467 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579327 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579654 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.579883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.579831 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.580216 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.580182 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.581780 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.581756 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.582058 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.582038 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.587409 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.587382 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.587607 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.587589 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslck\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:19.741723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:19.741688 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:20.060284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.060252 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 22:58:20.061602 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:58:20.061570 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5e8124_afe2_4e38_b3e1_4e63cb2a7aba.slice/crio-cc928890fbffb86babef82cae46788bc1ddb58bfa79bb9ace9e99dea6abf0a8a WatchSource:0}: Error finding container cc928890fbffb86babef82cae46788bc1ddb58bfa79bb9ace9e99dea6abf0a8a: Status 404 returned error can't find the container with id cc928890fbffb86babef82cae46788bc1ddb58bfa79bb9ace9e99dea6abf0a8a Nov 01 22:58:20.066184 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.066136 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:20.066276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.066221 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:20.066276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.066264 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:20.215771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.215717 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" event={"ID":"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba","Type":"ContainerStarted","Data":"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2"} Nov 01 22:58:20.216008 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.215788 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" event={"ID":"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba","Type":"ContainerStarted","Data":"cc928890fbffb86babef82cae46788bc1ddb58bfa79bb9ace9e99dea6abf0a8a"} Nov 01 22:58:20.239830 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.239767 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podStartSLOduration=1.239746574 podStartE2EDuration="1.239746574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:20.237079285 +0000 UTC m=+2070.013066115" watchObservedRunningTime="2026-11-01 22:58:20.239746574 +0000 UTC m=+2070.015733403" Nov 01 22:58:20.747878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:20.743997 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:21.756876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:21.755250 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:21.756876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:21.755330 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:22.742789 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:22.742195 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:22.743425 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:22.742257 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:23.747877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:23.747369 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:23.747877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:23.747440 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:24.744070 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:24.744025 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:24.745476 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:24.745040 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:25.747387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:25.743192 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:25.747387 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:25.743254 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:26.311812 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.311780 5778 generic.go:356] "Generic (PLEG): container finished" podID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerID="640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27" exitCode=0 Nov 01 22:58:26.312047 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.311875 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerDied","Data":"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27"} Nov 01 22:58:26.312385 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.312367 5778 scope.go:122] "RemoveContainer" containerID="640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27" Nov 01 22:58:26.314360 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.314330 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:26.314477 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.314415 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:26.314530 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.314479 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:58:26.743059 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.743013 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 22:58:26.743274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:26.743106 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 22:58:27.356364 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.356322 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerStarted","Data":"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f"} Nov 01 22:58:27.402755 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.402729 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:58:27.430357 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.430316 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:58:27.431874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.431832 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:27.432032 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.431920 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:27.754640 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.754605 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:27.755865 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.755836 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:27.756517 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:27.756496 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 22:58:28.434307 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:28.434268 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:28.434964 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:28.434930 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:29.437820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:29.437219 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:29.437820 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:29.437293 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:30.451889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:30.451234 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:30.451889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:30.451310 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:31.454073 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:31.454037 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:31.454669 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:31.454636 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:32.457961 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:32.457921 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:32.458753 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:32.458718 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:33.437876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:33.431243 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:58:33.437876 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:33.431341 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 01 22:58:34.435934 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:34.435903 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:58:34.437098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:34.437073 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 22:58:36.130287 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.130253 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:58:36.131111 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.131074 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" containerID="cri-o://f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a" gracePeriod=30 Nov 01 22:58:36.177644 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.177609 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:58:36.178104 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.178067 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" containerID="cri-o://7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f" gracePeriod=30 Nov 01 22:58:36.190910 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.190877 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:58:36.191095 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:36.191070 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" containerID="cri-o://c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa" gracePeriod=30 Nov 01 22:58:41.134687 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.134651 5778 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:58:41.135170 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.134725 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:58:41.543904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.541660 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:58:41.560896 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.558843 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660077 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660237 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660295 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660370 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660430 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660464 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660497 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660550 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660592 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660642 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660673 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.660761 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661117 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661208 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661274 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661383 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661491 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\") pod \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\" (UID: \"15b4df03-ee7f-4182-9e99-c8c45dcffba8\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.661571 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\") pod \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\" (UID: \"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3\") " Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.662844 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.663103 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.663327 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.663544 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.665055 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.663774 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.666672 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.667040 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.667355 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.684365 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.689677 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.696889 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.694323 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.699906 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "kube-api-access-5ddst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.702665 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.709096 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.709272 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.709531 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.715404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.713406 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f" pod "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" (UID: "860f831e-caa5-4dca-8fe8-a1fed6bdf1d3"). InnerVolumeSpecName "kube-api-access-v2b6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.716933 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.716842 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo" pod "15b4df03-ee7f-4182-9e99-c8c45dcffba8" (UID: "15b4df03-ee7f-4182-9e99-c8c45dcffba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.725636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.724717 5778 generic.go:356] "Generic (PLEG): container finished" podID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerID="c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa" exitCode=0 Nov 01 22:58:41.725636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.724751 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:58:41.725636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.724825 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" event={"ID":"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78","Type":"ContainerDied","Data":"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa"} Nov 01 22:58:41.725636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.724897 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" event={"ID":"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78","Type":"ContainerDied","Data":"bbb35b95ce5fef20c25129de0c6615228c6fd01cb4f7656bf6c83f6f4a65d8df"} Nov 01 22:58:41.725636 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.724932 5778 scope.go:122] "RemoveContainer" containerID="c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa" Nov 01 22:58:41.729644 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.729620 5778 generic.go:356] "Generic (PLEG): container finished" podID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerID="7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f" exitCode=0 Nov 01 22:58:41.730001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.729983 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" Nov 01 22:58:41.730723 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.730703 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" event={"ID":"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3","Type":"ContainerDied","Data":"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f"} Nov 01 22:58:41.730955 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.730907 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29" event={"ID":"860f831e-caa5-4dca-8fe8-a1fed6bdf1d3","Type":"ContainerDied","Data":"eba4982a02d2008c9b6102e563ef87e8878a926f238a46f74b5833f60b2415a2"} Nov 01 22:58:41.732436 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.732411 5778 generic.go:356] "Generic (PLEG): container finished" podID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerID="f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a" exitCode=0 Nov 01 22:58:41.732559 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.732490 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" Nov 01 22:58:41.732984 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.732960 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" event={"ID":"15b4df03-ee7f-4182-9e99-c8c45dcffba8","Type":"ContainerDied","Data":"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a"} Nov 01 22:58:41.733103 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.733054 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6" event={"ID":"15b4df03-ee7f-4182-9e99-c8c45dcffba8","Type":"ContainerDied","Data":"39c411b9797dfce6ff5fcab4f993bc8901f163905c2f22036efa5ebfcd5c3878"} Nov 01 22:58:41.751470 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.751443 5778 scope.go:122] "RemoveContainer" containerID="c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa" Nov 01 22:58:41.751992 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:41.751966 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa\": container with ID starting with c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa not found: ID does not exist" containerID="c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa" Nov 01 22:58:41.755077 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.752000 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa"} err="failed to get container status \"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa\": rpc error: code = NotFound desc = could not find container \"c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa\": container with ID starting with c7158109787fc5360a11c050a08e45c2a3be04271467d39cdce892b16c1dfaaa not found: ID does not exist" Nov 01 22:58:41.755235 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.755084 5778 scope.go:122] "RemoveContainer" containerID="7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f" Nov 01 22:58:41.763764 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763742 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763769 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763783 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763797 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddst\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-kube-api-access-5ddst\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763815 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763829 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763842 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763873 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763886 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763899 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.763917 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763912 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763927 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2b6f\" (UniqueName: \"kubernetes.io/projected/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-kube-api-access-v2b6f\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763941 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763954 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763967 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b4df03-ee7f-4182-9e99-c8c45dcffba8-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763980 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.763995 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.764319 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.764010 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b4df03-ee7f-4182-9e99-c8c45dcffba8-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.772988 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.772970 5778 scope.go:122] "RemoveContainer" containerID="7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f" Nov 01 22:58:41.773345 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:41.773319 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f\": container with ID starting with 7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f not found: ID does not exist" containerID="7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f" Nov 01 22:58:41.773417 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.773358 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f"} err="failed to get container status \"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f\": rpc error: code = NotFound desc = could not find container \"7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f\": container with ID starting with 7e03224ba5802da84934058c1b90b45c299d3219c69521aec76e4845cc68f16f not found: ID does not exist" Nov 01 22:58:41.773417 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.773381 5778 scope.go:122] "RemoveContainer" containerID="f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a" Nov 01 22:58:41.783510 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.783422 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:58:41.788567 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.788484 5778 scope.go:122] "RemoveContainer" containerID="f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a" Nov 01 22:58:41.788958 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.788933 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-xk4r6"] Nov 01 22:58:41.789148 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:41.788960 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a\": container with ID starting with f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a not found: ID does not exist" containerID="f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a" Nov 01 22:58:41.789297 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.789266 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a"} err="failed to get container status \"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a\": rpc error: code = NotFound desc = could not find container \"f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a\": container with ID starting with f93db80d9dfbfcc39616370d6f15918777bc401229fba456366386e5a4332f7a not found: ID does not exist" Nov 01 22:58:41.797924 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.795819 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:58:41.801139 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.801115 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-zfs29"] Nov 01 22:58:41.864546 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864516 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.864659 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864616 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.864749 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864735 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.864786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864764 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.864786 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864774 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.864935 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864891 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.865001 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.864981 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.865087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865020 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.865087 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865062 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.865498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865086 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.865498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865165 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\") pod \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\" (UID: \"ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78\") " Nov 01 22:58:41.865498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865346 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.865498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865367 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.865498 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.865440 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.866209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.866186 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.866285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.866217 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.866285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.866234 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.866285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.866247 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.866285 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.866260 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.867365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.867333 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.867518 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.867498 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "kube-api-access-7bmwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.867701 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.867681 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.867763 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.867744 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo" pod "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" (UID: "ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.967404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.967345 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.967404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.967369 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.967404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.967378 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bmwt\" (UniqueName: \"kubernetes.io/projected/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-kube-api-access-7bmwt\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.967404 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:41.967389 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:42.447223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:42.445098 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" path="/var/lib/kubelet/pods/15b4df03-ee7f-4182-9e99-c8c45dcffba8/volumes" Nov 01 22:58:42.447223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:42.445837 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" path="/var/lib/kubelet/pods/860f831e-caa5-4dca-8fe8-a1fed6bdf1d3/volumes" Nov 01 22:58:42.745799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:42.745772 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5" Nov 01 22:58:42.767930 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:42.767903 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:58:42.772015 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:42.771992 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-t6zp5"] Nov 01 22:58:44.377220 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:44.377181 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" path="/var/lib/kubelet/pods/ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78/volumes" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.512283 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513516 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513543 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513565 5778 cpu_manager.go:414] "containerMap: removing container" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513572 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513642 5778 cpu_manager.go:414] "containerMap: removing container" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513651 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513824 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b4df03-ee7f-4182-9e99-c8c45dcffba8" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513842 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4da1ec-9d9e-4235-aed8-8c2edf6d2a78" containerName="istio-proxy" Nov 01 22:58:47.515878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.513872 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="860f831e-caa5-4dca-8fe8-a1fed6bdf1d3" containerName="istio-proxy" Nov 01 22:58:47.550972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.550467 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 22:58:47.550972 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.550591 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.563914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.558555 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-788d7b\"/\"default-dockercfg-rwh9s\"" Nov 01 22:58:47.563914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.558715 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-788d7b\"/\"auth-test-model-788d7b-kserve-self-signed-certs\"" Nov 01 22:58:47.563914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.558790 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-788d7b\"/\"kube-root-ca.crt\"" Nov 01 22:58:47.563914 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.558841 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-788d7b\"/\"openshift-service-ca.crt\"" Nov 01 22:58:47.652156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.650043 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5hc\" (UniqueName: \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.652693 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.652597 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.652947 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.652927 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.653375 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.653355 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.653652 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.653633 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.654041 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.654021 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.755797 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.755761 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.756060 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.755902 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5hc\" (UniqueName: \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.756365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.756332 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.756492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.756391 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.756805 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.756777 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.757156 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.757086 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.757474 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.757444 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.757900 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.757877 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.758276 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.758249 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.761169 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.760813 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.761361 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.761338 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.764614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.764586 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5hc\" (UniqueName: \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\") pod \"auth-test-model-788d7b-kserve-64f569b89c-sfpmx\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:47.910442 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:47.910164 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:48.411567 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:58:48.411533 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274ef331_6979_424e_a3e8_42f08379bd61.slice/crio-c06a371a7d8c414fc88d5583a8793cb300b2f67f70a2e9c9c233d9131ad09758 WatchSource:0}: Error finding container c06a371a7d8c414fc88d5583a8793cb300b2f67f70a2e9c9c233d9131ad09758: Status 404 returned error can't find the container with id c06a371a7d8c414fc88d5583a8793cb300b2f67f70a2e9c9c233d9131ad09758 Nov 01 22:58:48.414321 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:48.414291 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 22:58:48.778008 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:58:48.777920 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:58:48.842346 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:48.842310 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" event={"ID":"274ef331-6979-424e-a3e8-42f08379bd61","Type":"ContainerStarted","Data":"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21"} Nov 01 22:58:48.842544 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:48.842363 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" event={"ID":"274ef331-6979-424e-a3e8-42f08379bd61","Type":"ContainerStarted","Data":"c06a371a7d8c414fc88d5583a8793cb300b2f67f70a2e9c9c233d9131ad09758"} Nov 01 22:58:48.861413 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:48.861357 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" podStartSLOduration=1.861343405 podStartE2EDuration="1.861343405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:48.859601711 +0000 UTC m=+2098.635588540" watchObservedRunningTime="2026-11-01 22:58:48.861343405 +0000 UTC m=+2098.637330233" Nov 01 22:58:48.912024 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:48.911993 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:50.792960 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:50.792845 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:58:50.804253 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:50.804211 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 22:58:57.911554 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:57.911516 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:57.926257 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:57.926229 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:58:57.965845 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:58:57.965815 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 22:59:08.026113 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.026074 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:59:08.027003 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.026948 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" containerID="cri-o://a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0" gracePeriod=30 Nov 01 22:59:08.600448 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.600414 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 22:59:08.612310 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.612285 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 22:59:08.614492 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.614467 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.698880 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.663349 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d73b25\"/\"default-dockercfg-64n7f\"" Nov 01 22:59:08.751967 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.702112 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d73b25\"/\"openshift-service-ca.crt\"" Nov 01 22:59:08.752209 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.704760 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d73b25\"/\"kube-root-ca.crt\"" Nov 01 22:59:08.767931 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.735567 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d73b25\"/\"auth-test-model-d73b25-kserve-self-signed-certs\"" Nov 01 22:59:08.822915 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.812518 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn9k\" (UniqueName: \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.823284 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.823257 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.823517 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.823502 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.824829 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.824808 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.837190 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.837154 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.837358 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.837234 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939332 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939412 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939607 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn9k\" (UniqueName: \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939685 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939759 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.939874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.939809 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.940466 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.940439 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.942025 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.941137 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.942392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.942371 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:08.972883 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:08.969313 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:09.010065 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:09.005980 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:09.041565 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:09.039310 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn9k\" (UniqueName: \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\") pod \"auth-test-model-d73b25-kserve-7c7bf5df-kzbwc\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:09.139740 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:09.139704 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:09.458733 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:09.458698 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 22:59:09.460333 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:59:09.460304 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c90268_84cd_494d_ae67_4e3f3d7ec094.slice/crio-05ea5b439e7ead8bf4f2315a7f96e1e185e9403c382110e929f0988f0a89a3b2 WatchSource:0}: Error finding container 05ea5b439e7ead8bf4f2315a7f96e1e185e9403c382110e929f0988f0a89a3b2: Status 404 returned error can't find the container with id 05ea5b439e7ead8bf4f2315a7f96e1e185e9403c382110e929f0988f0a89a3b2 Nov 01 22:59:10.083011 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:10.082972 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" event={"ID":"a0c90268-84cd-494d-ae67-4e3f3d7ec094","Type":"ContainerStarted","Data":"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d"} Nov 01 22:59:10.083428 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:10.083029 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" event={"ID":"a0c90268-84cd-494d-ae67-4e3f3d7ec094","Type":"ContainerStarted","Data":"05ea5b439e7ead8bf4f2315a7f96e1e185e9403c382110e929f0988f0a89a3b2"} Nov 01 22:59:10.109517 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:10.109454 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" podStartSLOduration=2.109438416 podStartE2EDuration="2.109438416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:10.106763671 +0000 UTC m=+2119.882750501" watchObservedRunningTime="2026-11-01 22:59:10.109438416 +0000 UTC m=+2119.885425244" Nov 01 22:59:10.140688 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:10.140652 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:13.367782 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.367754 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:59:13.438566 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.438529 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.438783 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.438688 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.438879 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.438787 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.438966 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.438950 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439048 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439032 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439109 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439074 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439165 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439124 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439223 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439174 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439348 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439335 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\") pod \"459bfc6b-e188-4475-9516-926afc86f1d2\" (UID: \"459bfc6b-e188-4475-9516-926afc86f1d2\") " Nov 01 22:59:13.439454 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439425 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.439673 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439651 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.439974 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.439948 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.440157 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.440135 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.440236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.440163 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.440236 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.440178 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.440556 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.440487 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.442365 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.442307 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:13.446664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.445216 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.446664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.445251 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:59:13.446664 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.445370 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:13.450130 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.447116 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg" pod "459bfc6b-e188-4475-9516-926afc86f1d2" (UID: "459bfc6b-e188-4475-9516-926afc86f1d2"). InnerVolumeSpecName "kube-api-access-h28mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553516 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553550 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553563 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28mg\" (UniqueName: \"kubernetes.io/projected/459bfc6b-e188-4475-9516-926afc86f1d2-kube-api-access-h28mg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553575 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459bfc6b-e188-4475-9516-926afc86f1d2-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553589 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459bfc6b-e188-4475-9516-926afc86f1d2-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.558874 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:13.553600 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459bfc6b-e188-4475-9516-926afc86f1d2-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:14.034736 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.034705 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf"] Nov 01 22:59:14.041614 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.041582 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab5dca57-dxmsf"] Nov 01 22:59:14.118392 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.118353 5778 generic.go:356] "Generic (PLEG): container finished" podID="459bfc6b-e188-4475-9516-926afc86f1d2" containerID="a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0" exitCode=0 Nov 01 22:59:14.118666 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.118643 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" event={"ID":"459bfc6b-e188-4475-9516-926afc86f1d2","Type":"ContainerDied","Data":"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0"} Nov 01 22:59:14.118769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.118675 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" Nov 01 22:59:14.118769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.118710 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf" event={"ID":"459bfc6b-e188-4475-9516-926afc86f1d2","Type":"ContainerDied","Data":"8d1e0da18868d0b128998be5b01f7ea071f998cd741d0fa1dfb537e8dab2a66a"} Nov 01 22:59:14.118769 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.118739 5778 scope.go:122] "RemoveContainer" containerID="a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0" Nov 01 22:59:14.157122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.153908 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:59:14.157122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.155607 5778 scope.go:122] "RemoveContainer" containerID="a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0" Nov 01 22:59:14.157122 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:59:14.156109 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0\": container with ID starting with a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0 not found: ID does not exist" containerID="a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0" Nov 01 22:59:14.157122 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.156143 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0"} err="failed to get container status \"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0\": rpc error: code = NotFound desc = could not find container \"a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0\": container with ID starting with a9f2e31e0c5d9c9c0aefc846c430db38c65941175033491ad5f3297034cf99a0 not found: ID does not exist" Nov 01 22:59:14.160098 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.160071 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-nwgcf"] Nov 01 22:59:14.388746 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.388662 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274be1a9-ad76-479a-8dca-d9bc815c0a17" path="/var/lib/kubelet/pods/274be1a9-ad76-479a-8dca-d9bc815c0a17/volumes" Nov 01 22:59:14.389677 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:14.389647 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" path="/var/lib/kubelet/pods/459bfc6b-e188-4475-9516-926afc86f1d2/volumes" Nov 01 22:59:18.799268 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:59:18.799225 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:59:19.140099 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:19.139997 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:19.153502 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:19.153476 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:19.301274 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:19.301229 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 22:59:23.269034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.268962 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 22:59:23.270145 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.270119 5778 cpu_manager.go:414] "containerMap: removing container" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" Nov 01 22:59:23.270145 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.270142 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" Nov 01 22:59:23.270359 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.270305 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="459bfc6b-e188-4475-9516-926afc86f1d2" containerName="istio-proxy" Nov 01 22:59:23.276906 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.276841 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 22:59:23.277019 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.276958 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.284761 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.284709 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"openshift-service-ca.crt\"" Nov 01 22:59:23.284908 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.284811 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"auth-test-model-7366bf-kserve-self-signed-certs\"" Nov 01 22:59:23.285034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.284936 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"default-dockercfg-l55dr\"" Nov 01 22:59:23.285034 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.285016 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"kube-root-ca.crt\"" Nov 01 22:59:23.429875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.429248 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.429875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.429438 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.429875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.429570 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.429875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.429722 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.462904 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.458649 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7m4p\" (UniqueName: \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.463661 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.463628 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.574641 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.574521 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.575062 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.575041 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.575320 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.575293 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.587573 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.587811 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.588154 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.588202 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m4p\" (UniqueName: \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.591474 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.593872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.592923 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.619872 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.611504 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.642487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.637890 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.642487 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.638489 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7m4p\" (UniqueName: \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\") pod \"auth-test-model-7366bf-kserve-88cb49ddf-lshtr\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:23.916088 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:23.916055 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:24.226771 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:24.226741 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 22:59:24.342192 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:24.342162 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" event={"ID":"e4cce74b-0093-4df5-b5e7-200b8da48ced","Type":"ContainerStarted","Data":"1c901d223a7f5e4e0616635faf43e3deb1a99a697980de74d132715af57e7daa"} Nov 01 22:59:25.051878 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:25.050967 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94"] Nov 01 22:59:25.061332 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:25.059395 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f6554-l4n94"] Nov 01 22:59:25.354459 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:25.354349 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" event={"ID":"e4cce74b-0093-4df5-b5e7-200b8da48ced","Type":"ContainerStarted","Data":"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f"} Nov 01 22:59:25.372751 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:25.372707 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" podStartSLOduration=2.3726928689999998 podStartE2EDuration="2.372692869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:25.371564704 +0000 UTC m=+2135.147551532" watchObservedRunningTime="2026-11-01 22:59:25.372692869 +0000 UTC m=+2135.148679701" Nov 01 22:59:25.916302 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:25.916270 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:26.376199 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:26.376123 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c" path="/var/lib/kubelet/pods/fc8b1eaa-3d6b-40b1-aecb-bd2ef376465c/volumes" Nov 01 22:59:33.916529 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:33.916367 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:33.934799 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:33.934771 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:34.450473 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:34.450442 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 22:59:40.996887 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:40.993147 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 22:59:41.020994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.009325 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 22:59:41.020994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.009440 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.020994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.017015 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"default-dockercfg-866mq\"" Nov 01 22:59:41.020994 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.017578 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"auth-test-model-075771-kserve-self-signed-certs\"" Nov 01 22:59:41.085672 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.085630 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"kube-root-ca.crt\"" Nov 01 22:59:41.087163 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.086049 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"openshift-service-ca.crt\"" Nov 01 22:59:41.162873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162322 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.162873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162373 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.162873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162587 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.162873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162729 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.162873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162842 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.163349 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.162905 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtvg\" (UniqueName: \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.265951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.265003 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.265951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.265242 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.265951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.265374 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.265951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.265463 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.265951 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.265503 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtvg\" (UniqueName: \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.278873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.276323 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.278873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.276796 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.278873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.277283 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.278873 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.277711 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.311875 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.310587 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtvg\" (UniqueName: \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.367877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.354522 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.367877 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.362795 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\") pod \"auth-test-model-075771-kserve-95bdd65df-rbgk9\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.647716 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.647615 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:41.996759 ip-10-0-141-223 kubenswrapper[5778]: W1101 22:59:41.996725 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0462f7f_2b3b_46af_9638_b6010c9e7e6d.slice/crio-8ec535fa07bf73c5687e97e5b17ca43d5db58afd3787ef5e2f6bb52c6d532b40 WatchSource:0}: Error finding container 8ec535fa07bf73c5687e97e5b17ca43d5db58afd3787ef5e2f6bb52c6d532b40: Status 404 returned error can't find the container with id 8ec535fa07bf73c5687e97e5b17ca43d5db58afd3787ef5e2f6bb52c6d532b40 Nov 01 22:59:41.999018 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:41.998984 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 22:59:42.517939 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:42.517904 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" event={"ID":"d0462f7f-2b3b-46af-9638-b6010c9e7e6d","Type":"ContainerStarted","Data":"192bf58fd814f56f4bddbfc74ccc69e737355f5443369f7a5f16079612bb80e3"} Nov 01 22:59:42.518133 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:42.517949 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" event={"ID":"d0462f7f-2b3b-46af-9638-b6010c9e7e6d","Type":"ContainerStarted","Data":"8ec535fa07bf73c5687e97e5b17ca43d5db58afd3787ef5e2f6bb52c6d532b40"} Nov 01 22:59:42.536452 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:42.536401 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" podStartSLOduration=2.5363890209999997 podStartE2EDuration="2.536389021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:42.535331506 +0000 UTC m=+2152.311318334" watchObservedRunningTime="2026-11-01 22:59:42.536389021 +0000 UTC m=+2152.312375850" Nov 01 22:59:42.670881 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:42.660969 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:48.775162 ip-10-0-141-223 kubenswrapper[5778]: E1101 22:59:48.775123 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 22:59:51.648602 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:51.648546 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:51.662308 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:51.662280 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:52.657461 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:52.657412 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 22:59:55.014420 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:55.014374 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 22:59:57.763322 ip-10-0-141-223 kubenswrapper[5778]: I1101 22:59:57.763129 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 23:00:00.215467 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.215260 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:00.228748 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.228721 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.230675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.230636 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:00.245232 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.245197 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:00.247408 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.245751 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.248999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.248976 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:00:00.254367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.251739 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.254367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.251776 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:00.272585 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.272558 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:00.284056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.284031 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:00.286019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.285995 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:00.291418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.291393 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.301960 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.301937 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:00.307786 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.307752 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.313127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.310789 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:00.352593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.352561 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:00.368935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.368796 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.369114 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369087 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccll\" (UniqueName: \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.369170 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369139 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k7j\" (UniqueName: \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.369272 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369254 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.369458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369439 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2x7k\" (UniqueName: \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.369512 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369497 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.369777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.369757 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:00.380106 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.380080 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.442459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.442421 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:00.442635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.442590 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:00.455843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.451901 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.475296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475222 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2x7k\" (UniqueName: \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.475296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475288 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.475513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475354 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.475513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475415 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnxv\" (UniqueName: \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.475513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475482 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbln7\" (UniqueName: \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.475678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475594 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.475678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475647 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccll\" (UniqueName: \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.475896 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.475844 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:00.476230 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.476199 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k7j\" (UniqueName: \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.476743 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.476714 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.477058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477029 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.477155 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477110 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.477554 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477501 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.478108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477588 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.478108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477732 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.478108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.477739 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.494380 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.494268 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:00.494524 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.494499 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.497369 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.497348 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:00.499072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.499045 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2x7k\" (UniqueName: \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.499899 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.499874 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccll\" (UniqueName: \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.500175 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.500155 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k7j\" (UniqueName: \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.578594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.578567 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:00.579213 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579190 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.579290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579247 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnxv\" (UniqueName: \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.579290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579288 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbln7\" (UniqueName: \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.579400 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579384 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmx7d\" (UniqueName: \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.579473 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579432 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.579675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579652 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.579744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579692 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.579744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579735 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.579828 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579762 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7sz\" (UniqueName: \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.579828 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.579800 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.580241 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.580217 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.580365 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.580347 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.580428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.580410 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.591800 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.591776 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbln7\" (UniqueName: \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\") pod \"maas-api-key-cleanup-29892900-npbx2\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.592178 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.592156 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.592393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.592358 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnxv\" (UniqueName: \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.593797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.593775 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:00.605359 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.605339 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:00.624276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.624241 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:00.641003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.640579 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:00.685914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.682333 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.685914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.682536 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmx7d\" (UniqueName: \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.685914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.682768 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.685914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.682812 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7sz\" (UniqueName: \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.685914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.684021 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.686546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.686484 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.694508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.694456 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7sz\" (UniqueName: \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.739198 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.739085 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmx7d\" (UniqueName: \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.758183 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.757682 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:00.794176 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.792476 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:00.858170 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.857917 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:00.898045 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:00.897888 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:00.906751 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:00:00.906718 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bf8dd8_f080_49fd_983f_64a7b6dd235c.slice/crio-364ab5616d3ed89d8f773f8eae655847aec49f145e3fd5e0bf965f3c2d69ee6a WatchSource:0}: Error finding container 364ab5616d3ed89d8f773f8eae655847aec49f145e3fd5e0bf965f3c2d69ee6a: Status 404 returned error can't find the container with id 364ab5616d3ed89d8f773f8eae655847aec49f145e3fd5e0bf965f3c2d69ee6a Nov 01 23:00:01.168389 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:00:01.168359 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a106c5_77bf_4556_9fae_288669ebf91b.slice/crio-56753090b6e02b9faacc5bdee802045ffab22252a5791f81d2939b32762c7545 WatchSource:0}: Error finding container 56753090b6e02b9faacc5bdee802045ffab22252a5791f81d2939b32762c7545: Status 404 returned error can't find the container with id 56753090b6e02b9faacc5bdee802045ffab22252a5791f81d2939b32762c7545 Nov 01 23:00:01.171804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.171770 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:01.238798 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.238764 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:01.247581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.247555 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:01.312431 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.312404 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:01.382460 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.382436 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:01.397818 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.397794 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:01.413310 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.413279 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:01.776438 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.776401 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerStarted","Data":"792b2744d94f6bf7e9df81c6828f5d2a662d669ad4ac430be58d18fe6b23cf7e"} Nov 01 23:00:01.779371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.779332 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerStarted","Data":"d96b4a6159d7c467d8f185cc0b6b1e52cf88cd01bb0f66ff6c1aefe040682a05"} Nov 01 23:00:01.803886 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.802374 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerStarted","Data":"fd1436f6dc2e42969c55e1db2152294fd754e205409975a0cc789a1844393c3b"} Nov 01 23:00:01.823878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.817405 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerStarted","Data":"a1ae22828a9b7e1843db77ffdc754850828b091b4e0e1e889ab4d1ae610cb604"} Nov 01 23:00:01.836164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.833898 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerStarted","Data":"90d50d9efbf481c3a5f2701bd344350686692fa1fe85005edd84feec3331ef86"} Nov 01 23:00:01.836164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.833965 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerStarted","Data":"364ab5616d3ed89d8f773f8eae655847aec49f145e3fd5e0bf965f3c2d69ee6a"} Nov 01 23:00:01.849532 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.849497 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerStarted","Data":"81f6964854e3b98ac0fd86b332fa288c0c8af4f6502e1475f74dd5d5d7dd2fee"} Nov 01 23:00:01.851865 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.851819 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerStarted","Data":"e477e6705e88a14b03f3a653e16f044068c7041079ab98566a9201a27111cd75"} Nov 01 23:00:01.854269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.854230 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerStarted","Data":"c3f4f4d7bbaa52fe727c64ad7daf5b5d7d4a42a884f80e1322dc4a73158fcab7"} Nov 01 23:00:01.854527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.854491 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerStarted","Data":"56753090b6e02b9faacc5bdee802045ffab22252a5791f81d2939b32762c7545"} Nov 01 23:00:01.863680 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.861840 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" podStartSLOduration=1.86182767 podStartE2EDuration="1.86182767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:01.859324332 +0000 UTC m=+2171.635311162" watchObservedRunningTime="2026-11-01 23:00:01.86182767 +0000 UTC m=+2171.637814499" Nov 01 23:00:01.880103 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:01.880053 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" podStartSLOduration=1.880030175 podStartE2EDuration="1.880030175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:01.874628976 +0000 UTC m=+2171.650615806" watchObservedRunningTime="2026-11-01 23:00:01.880030175 +0000 UTC m=+2171.656017004" Nov 01 23:00:02.865603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.865567 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerStarted","Data":"8ab3238d3058fc261eecadddbf5b7d3efc55614fa7d079960ecbbdb752c12075"} Nov 01 23:00:02.867358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.867331 5778 generic.go:356] "Generic (PLEG): container finished" podID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerID="e553bd0002c6aae7221cfc9b713c5411371ff749e015b4b4c87c51b5cb073797" exitCode=60 Nov 01 23:00:02.867456 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.867412 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerDied","Data":"e553bd0002c6aae7221cfc9b713c5411371ff749e015b4b4c87c51b5cb073797"} Nov 01 23:00:02.867663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.867647 5778 scope.go:122] "RemoveContainer" containerID="e553bd0002c6aae7221cfc9b713c5411371ff749e015b4b4c87c51b5cb073797" Nov 01 23:00:02.869760 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.869382 5778 generic.go:356] "Generic (PLEG): container finished" podID="54209855-7b21-4953-94cd-2035c2506f6c" containerID="b0b2e21eb124e0e3301c10a6b62bc21e11a971ecd893021b5a0947c635123f30" exitCode=60 Nov 01 23:00:02.869760 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.869485 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerDied","Data":"b0b2e21eb124e0e3301c10a6b62bc21e11a971ecd893021b5a0947c635123f30"} Nov 01 23:00:02.869760 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.869693 5778 scope.go:122] "RemoveContainer" containerID="b0b2e21eb124e0e3301c10a6b62bc21e11a971ecd893021b5a0947c635123f30" Nov 01 23:00:02.871269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.871243 5778 generic.go:356] "Generic (PLEG): container finished" podID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerID="90d50d9efbf481c3a5f2701bd344350686692fa1fe85005edd84feec3331ef86" exitCode=60 Nov 01 23:00:02.871368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.871346 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerDied","Data":"90d50d9efbf481c3a5f2701bd344350686692fa1fe85005edd84feec3331ef86"} Nov 01 23:00:02.871618 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.871601 5778 scope.go:122] "RemoveContainer" containerID="90d50d9efbf481c3a5f2701bd344350686692fa1fe85005edd84feec3331ef86" Nov 01 23:00:02.878463 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.878441 5778 generic.go:356] "Generic (PLEG): container finished" podID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerID="7845231c52e3feae422d619e60023734bf00127f8c7a6de4f79290c0f878984e" exitCode=60 Nov 01 23:00:02.878574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.878554 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerDied","Data":"7845231c52e3feae422d619e60023734bf00127f8c7a6de4f79290c0f878984e"} Nov 01 23:00:02.878740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.878722 5778 scope.go:122] "RemoveContainer" containerID="7845231c52e3feae422d619e60023734bf00127f8c7a6de4f79290c0f878984e" Nov 01 23:00:02.880643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.880623 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b061025-9230-460b-9d8d-0ab15676b77c" containerID="884bc478a9a166259ca7fc0dd978db57750bb4f54d1b163433824ffe10e74029" exitCode=60 Nov 01 23:00:02.880728 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.880694 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerDied","Data":"884bc478a9a166259ca7fc0dd978db57750bb4f54d1b163433824ffe10e74029"} Nov 01 23:00:02.881110 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.881087 5778 scope.go:122] "RemoveContainer" containerID="884bc478a9a166259ca7fc0dd978db57750bb4f54d1b163433824ffe10e74029" Nov 01 23:00:02.884508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.884488 5778 generic.go:356] "Generic (PLEG): container finished" podID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerID="c3f4f4d7bbaa52fe727c64ad7daf5b5d7d4a42a884f80e1322dc4a73158fcab7" exitCode=60 Nov 01 23:00:02.884817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.884775 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerDied","Data":"c3f4f4d7bbaa52fe727c64ad7daf5b5d7d4a42a884f80e1322dc4a73158fcab7"} Nov 01 23:00:02.884956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.884926 5778 scope.go:122] "RemoveContainer" containerID="c3f4f4d7bbaa52fe727c64ad7daf5b5d7d4a42a884f80e1322dc4a73158fcab7" Nov 01 23:00:02.886743 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.886706 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" podStartSLOduration=2.886694743 podStartE2EDuration="2.886694743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.880916501 +0000 UTC m=+2172.656903346" watchObservedRunningTime="2026-11-01 23:00:02.886694743 +0000 UTC m=+2172.662681571" Nov 01 23:00:02.887702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:02.887672 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerStarted","Data":"5bf15fc7bb9691300a37dd6fb25446f1121a7a3e6613fa51857f5fb0da13973c"} Nov 01 23:00:03.076551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.074701 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" podStartSLOduration=3.074679977 podStartE2EDuration="3.074679977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.044017003 +0000 UTC m=+2172.820003832" watchObservedRunningTime="2026-11-01 23:00:03.074679977 +0000 UTC m=+2172.850666806" Nov 01 23:00:03.900373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.900258 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerStarted","Data":"89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69"} Nov 01 23:00:03.903975 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.903939 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerStarted","Data":"2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25"} Nov 01 23:00:03.906448 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.906418 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerStarted","Data":"5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884"} Nov 01 23:00:03.908521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.908490 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerStarted","Data":"d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41"} Nov 01 23:00:03.910418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.910393 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerStarted","Data":"5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7"} Nov 01 23:00:03.912662 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.912639 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerStarted","Data":"93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6"} Nov 01 23:00:03.914273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.914254 5778 generic.go:356] "Generic (PLEG): container finished" podID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerID="5bf15fc7bb9691300a37dd6fb25446f1121a7a3e6613fa51857f5fb0da13973c" exitCode=60 Nov 01 23:00:03.914411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.914384 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerDied","Data":"5bf15fc7bb9691300a37dd6fb25446f1121a7a3e6613fa51857f5fb0da13973c"} Nov 01 23:00:03.914536 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.914490 5778 scope.go:122] "RemoveContainer" containerID="5bf15fc7bb9691300a37dd6fb25446f1121a7a3e6613fa51857f5fb0da13973c" Nov 01 23:00:03.935096 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.935065 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerID="8ab3238d3058fc261eecadddbf5b7d3efc55614fa7d079960ecbbdb752c12075" exitCode=60 Nov 01 23:00:03.935223 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.935141 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerDied","Data":"8ab3238d3058fc261eecadddbf5b7d3efc55614fa7d079960ecbbdb752c12075"} Nov 01 23:00:03.935535 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.935514 5778 scope.go:122] "RemoveContainer" containerID="8ab3238d3058fc261eecadddbf5b7d3efc55614fa7d079960ecbbdb752c12075" Nov 01 23:00:03.936464 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.936326 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" podStartSLOduration=3.936310036 podStartE2EDuration="3.936310036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.918137558 +0000 UTC m=+2173.694124388" watchObservedRunningTime="2026-11-01 23:00:03.936310036 +0000 UTC m=+2173.712296870" Nov 01 23:00:03.936818 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.936785 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" podStartSLOduration=3.936778418 podStartE2EDuration="3.936778418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.933751147 +0000 UTC m=+2173.709737979" watchObservedRunningTime="2026-11-01 23:00:03.936778418 +0000 UTC m=+2173.712765315" Nov 01 23:00:03.957533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:03.957496 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" podStartSLOduration=3.9574846089999998 podStartE2EDuration="3.957484609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.95435143 +0000 UTC m=+2173.730338272" watchObservedRunningTime="2026-11-01 23:00:03.957484609 +0000 UTC m=+2173.733471438" Nov 01 23:00:04.050323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.050266 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" podStartSLOduration=4.050252219 podStartE2EDuration="4.050252219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.044952126 +0000 UTC m=+2173.820938968" watchObservedRunningTime="2026-11-01 23:00:04.050252219 +0000 UTC m=+2173.826239049" Nov 01 23:00:04.948016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.947979 5778 generic.go:356] "Generic (PLEG): container finished" podID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerID="89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69" exitCode=60 Nov 01 23:00:04.948470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.948029 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerDied","Data":"89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69"} Nov 01 23:00:04.948470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.948106 5778 scope.go:122] "RemoveContainer" containerID="e553bd0002c6aae7221cfc9b713c5411371ff749e015b4b4c87c51b5cb073797" Nov 01 23:00:04.948470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.948419 5778 scope.go:122] "RemoveContainer" containerID="89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69" Nov 01 23:00:04.948931 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.948899 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v_odh-ai-gateway-infra(8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" Nov 01 23:00:04.950453 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.950418 5778 generic.go:356] "Generic (PLEG): container finished" podID="54209855-7b21-4953-94cd-2035c2506f6c" containerID="2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25" exitCode=60 Nov 01 23:00:04.950586 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.950472 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerDied","Data":"2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25"} Nov 01 23:00:04.950822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.950795 5778 scope.go:122] "RemoveContainer" containerID="2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25" Nov 01 23:00:04.951248 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.951223 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h_odh-ai-gateway-infra(54209855-7b21-4953-94cd-2035c2506f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" podUID="54209855-7b21-4953-94cd-2035c2506f6c" Nov 01 23:00:04.952811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.952787 5778 generic.go:356] "Generic (PLEG): container finished" podID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerID="5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884" exitCode=60 Nov 01 23:00:04.952967 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.952907 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerDied","Data":"5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884"} Nov 01 23:00:04.953118 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.953098 5778 scope.go:122] "RemoveContainer" containerID="5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884" Nov 01 23:00:04.953400 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.953380 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p_odh-ai-gateway-infra(f9bf8dd8-f080-49fd-983f-64a7b6dd235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" Nov 01 23:00:04.955610 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.955586 5778 generic.go:356] "Generic (PLEG): container finished" podID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerID="d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41" exitCode=60 Nov 01 23:00:04.956424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.955724 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerDied","Data":"d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41"} Nov 01 23:00:04.956424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.955909 5778 scope.go:122] "RemoveContainer" containerID="d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41" Nov 01 23:00:04.956424 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.956202 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p_odh-ai-gateway-infra(5df4aab5-481c-4dce-a81e-08b5e79a0043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" Nov 01 23:00:04.959346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.958641 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b061025-9230-460b-9d8d-0ab15676b77c" containerID="5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7" exitCode=60 Nov 01 23:00:04.959346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.959049 5778 scope.go:122] "RemoveContainer" containerID="5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7" Nov 01 23:00:04.959346 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.959327 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd_odh-ai-gateway-infra(4b061025-9230-460b-9d8d-0ab15676b77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" Nov 01 23:00:04.959533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.959443 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerDied","Data":"5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7"} Nov 01 23:00:04.961953 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.961930 5778 generic.go:356] "Generic (PLEG): container finished" podID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerID="93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6" exitCode=60 Nov 01 23:00:04.962169 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.962039 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerDied","Data":"93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6"} Nov 01 23:00:04.962267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.962214 5778 scope.go:122] "RemoveContainer" containerID="93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6" Nov 01 23:00:04.962518 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:04.962491 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97_odh-ai-gateway-infra(e7a106c5-77bf-4556-9fae-288669ebf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" Nov 01 23:00:04.964714 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.964693 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerStarted","Data":"f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6"} Nov 01 23:00:04.968795 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.968772 5778 scope.go:122] "RemoveContainer" containerID="b0b2e21eb124e0e3301c10a6b62bc21e11a971ecd893021b5a0947c635123f30" Nov 01 23:00:04.968943 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.968810 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerStarted","Data":"4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177"} Nov 01 23:00:04.984259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:04.984237 5778 scope.go:122] "RemoveContainer" containerID="90d50d9efbf481c3a5f2701bd344350686692fa1fe85005edd84feec3331ef86" Nov 01 23:00:05.000497 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.000478 5778 scope.go:122] "RemoveContainer" containerID="7845231c52e3feae422d619e60023734bf00127f8c7a6de4f79290c0f878984e" Nov 01 23:00:05.033721 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.033699 5778 scope.go:122] "RemoveContainer" containerID="884bc478a9a166259ca7fc0dd978db57750bb4f54d1b163433824ffe10e74029" Nov 01 23:00:05.053127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.053104 5778 scope.go:122] "RemoveContainer" containerID="c3f4f4d7bbaa52fe727c64ad7daf5b5d7d4a42a884f80e1322dc4a73158fcab7" Nov 01 23:00:05.979985 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.979954 5778 scope.go:122] "RemoveContainer" containerID="2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25" Nov 01 23:00:05.980535 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.980328 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h_odh-ai-gateway-infra(54209855-7b21-4953-94cd-2035c2506f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" podUID="54209855-7b21-4953-94cd-2035c2506f6c" Nov 01 23:00:05.981982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.981957 5778 scope.go:122] "RemoveContainer" containerID="5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884" Nov 01 23:00:05.982454 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.982425 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p_odh-ai-gateway-infra(f9bf8dd8-f080-49fd-983f-64a7b6dd235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" Nov 01 23:00:05.983914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.983891 5778 scope.go:122] "RemoveContainer" containerID="d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41" Nov 01 23:00:05.984235 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.984210 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p_odh-ai-gateway-infra(5df4aab5-481c-4dce-a81e-08b5e79a0043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" Nov 01 23:00:05.985586 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.985530 5778 scope.go:122] "RemoveContainer" containerID="5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7" Nov 01 23:00:05.985842 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.985820 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd_odh-ai-gateway-infra(4b061025-9230-460b-9d8d-0ab15676b77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" Nov 01 23:00:05.987310 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.987285 5778 scope.go:122] "RemoveContainer" containerID="93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6" Nov 01 23:00:05.987596 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.987568 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97_odh-ai-gateway-infra(e7a106c5-77bf-4556-9fae-288669ebf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" Nov 01 23:00:05.989377 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.989353 5778 generic.go:356] "Generic (PLEG): container finished" podID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerID="f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6" exitCode=60 Nov 01 23:00:05.989471 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.989434 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerDied","Data":"f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6"} Nov 01 23:00:05.989531 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.989493 5778 scope.go:122] "RemoveContainer" containerID="5bf15fc7bb9691300a37dd6fb25446f1121a7a3e6613fa51857f5fb0da13973c" Nov 01 23:00:05.989682 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.989659 5778 scope.go:122] "RemoveContainer" containerID="f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6" Nov 01 23:00:05.990025 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.990001 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-npbx2_odh-ai-gateway-infra(e531e6a9-516c-46c3-98fe-4490852d7f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" Nov 01 23:00:05.991781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.991762 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerID="4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177" exitCode=60 Nov 01 23:00:05.991915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.991844 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerDied","Data":"4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177"} Nov 01 23:00:05.992123 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.992107 5778 scope.go:122] "RemoveContainer" containerID="4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177" Nov 01 23:00:05.992378 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.992361 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw_odh-ai-gateway-infra(4b4e7a4d-f6e8-430f-a48e-445b376201df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" Nov 01 23:00:05.993962 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:05.993941 5778 scope.go:122] "RemoveContainer" containerID="89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69" Nov 01 23:00:05.994226 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:05.994175 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v_odh-ai-gateway-infra(8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" Nov 01 23:00:06.057281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:06.057255 5778 scope.go:122] "RemoveContainer" containerID="8ab3238d3058fc261eecadddbf5b7d3efc55614fa7d079960ecbbdb752c12075" Nov 01 23:00:07.005777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:07.005750 5778 scope.go:122] "RemoveContainer" containerID="4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177" Nov 01 23:00:07.006328 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:07.006220 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw_odh-ai-gateway-infra(4b4e7a4d-f6e8-430f-a48e-445b376201df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" Nov 01 23:00:07.007681 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:07.007660 5778 scope.go:122] "RemoveContainer" containerID="f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6" Nov 01 23:00:07.008056 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:07.008027 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-npbx2_odh-ai-gateway-infra(e531e6a9-516c-46c3-98fe-4490852d7f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" Nov 01 23:00:08.057583 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:08.057549 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 23:00:10.054410 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.054371 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" containerID="cri-o://1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0" gracePeriod=45 Nov 01 23:00:10.465376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.465158 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477018 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477250 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477320 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477423 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\" (UniqueName: \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477633 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477834 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.477880 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.478109 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\") pod \"d0989540-2365-40ec-a84c-1e1263072037\" (UID: \"d0989540-2365-40ec-a84c-1e1263072037\") " Nov 01 23:00:10.478565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.478207 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:10.479158 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.479011 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:10.479158 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.479032 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:10.479158 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.479042 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:10.480834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.480806 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:10.481234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.481202 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:10.481505 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.481469 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh" pod "d0989540-2365-40ec-a84c-1e1263072037" (UID: "d0989540-2365-40ec-a84c-1e1263072037"). InnerVolumeSpecName "kube-api-access-j2sbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:10.581019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.580983 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0989540-2365-40ec-a84c-1e1263072037-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:10.581019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.581016 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2sbh\" (UniqueName: \"kubernetes.io/projected/d0989540-2365-40ec-a84c-1e1263072037-kube-api-access-j2sbh\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:10.581227 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:10.581031 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0989540-2365-40ec-a84c-1e1263072037-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:11.077024 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.076987 5778 generic.go:356] "Generic (PLEG): container finished" podID="d0989540-2365-40ec-a84c-1e1263072037" containerID="1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0" exitCode=0 Nov 01 23:00:11.077568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.077273 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" event={"ID":"d0989540-2365-40ec-a84c-1e1263072037","Type":"ContainerDied","Data":"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0"} Nov 01 23:00:11.077568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.077312 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" event={"ID":"d0989540-2365-40ec-a84c-1e1263072037","Type":"ContainerDied","Data":"77513b6314413fba98431ae381ef68be1d91fb3a289f496ba3d5b70b6dd5aa72"} Nov 01 23:00:11.077568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.077333 5778 scope.go:122] "RemoveContainer" containerID="1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0" Nov 01 23:00:11.077568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.077501 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" Nov 01 23:00:11.097267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.097237 5778 scope.go:122] "RemoveContainer" containerID="1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0" Nov 01 23:00:11.097702 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:11.097635 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0\": container with ID starting with 1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0 not found: ID does not exist" containerID="1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0" Nov 01 23:00:11.097702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.097668 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0"} err="failed to get container status \"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0\": rpc error: code = NotFound desc = could not find container \"1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0\": container with ID starting with 1fcb2bc75e5a993d1a77a73853b2f9894fa88b88278f001a49c9ef9c366c1ab0 not found: ID does not exist" Nov 01 23:00:11.114560 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.114535 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 23:00:11.121479 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.121456 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g"] Nov 01 23:00:11.415262 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:11.415202 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-021714/test-model-021714-kserve-8d45fb497-qbn5g" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded" Nov 01 23:00:12.387484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:12.387449 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0989540-2365-40ec-a84c-1e1263072037" path="/var/lib/kubelet/pods/d0989540-2365-40ec-a84c-1e1263072037/volumes" Nov 01 23:00:12.807148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:12.807099 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" containerID="cri-o://be85e823211c4db8c909ffad9448701b1ac1dd6fc20e63ea67f9a85e99b1f5a8" gracePeriod=45 Nov 01 23:00:13.098590 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.098563 5778 generic.go:356] "Generic (PLEG): container finished" podID="f89e722b-3638-42dc-afdc-76afef5f3159" containerID="be85e823211c4db8c909ffad9448701b1ac1dd6fc20e63ea67f9a85e99b1f5a8" exitCode=0 Nov 01 23:00:13.098715 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.098638 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" event={"ID":"f89e722b-3638-42dc-afdc-76afef5f3159","Type":"ContainerDied","Data":"be85e823211c4db8c909ffad9448701b1ac1dd6fc20e63ea67f9a85e99b1f5a8"} Nov 01 23:00:13.142435 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.142405 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 23:00:13.234737 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.234699 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\" (UniqueName: \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.235004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.234984 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.235097 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.235036 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\" (UniqueName: \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.235097 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.235085 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.235233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.235218 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.235296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.235287 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\") pod \"f89e722b-3638-42dc-afdc-76afef5f3159\" (UID: \"f89e722b-3638-42dc-afdc-76afef5f3159\") " Nov 01 23:00:13.239341 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.239313 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:13.243561 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.243535 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "kube-api-access-wbxcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:13.244067 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.244033 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:13.247426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.247321 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:13.247426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.247387 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:13.250250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.250221 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs" pod "f89e722b-3638-42dc-afdc-76afef5f3159" (UID: "f89e722b-3638-42dc-afdc-76afef5f3159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:13.337707 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337676 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:13.337707 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337707 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89e722b-3638-42dc-afdc-76afef5f3159-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:13.337917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337718 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:13.337917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337727 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:13.337917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337734 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89e722b-3638-42dc-afdc-76afef5f3159-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:13.337917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:13.337744 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxcn\" (UniqueName: \"kubernetes.io/projected/f89e722b-3638-42dc-afdc-76afef5f3159-kube-api-access-wbxcn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:14.069239 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.069195 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" probeResult="failure" output="Get \"https://10.132.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:00:14.110329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.110298 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" Nov 01 23:00:14.110329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.110319 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n" event={"ID":"f89e722b-3638-42dc-afdc-76afef5f3159","Type":"ContainerDied","Data":"753bac8baee0afe301896c78a152a6b11511766d0ec5bfc945c4e8d386a79db8"} Nov 01 23:00:14.110577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.110366 5778 scope.go:122] "RemoveContainer" containerID="be85e823211c4db8c909ffad9448701b1ac1dd6fc20e63ea67f9a85e99b1f5a8" Nov 01 23:00:14.146517 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.146483 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 23:00:14.153573 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.153544 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2f675d/test-model-2f675d-kserve-67d7bc44f9-tmg9n"] Nov 01 23:00:14.367579 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.367503 5778 scope.go:122] "RemoveContainer" containerID="5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7" Nov 01 23:00:14.385338 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:14.385303 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" path="/var/lib/kubelet/pods/f89e722b-3638-42dc-afdc-76afef5f3159/volumes" Nov 01 23:00:15.184879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:15.175171 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerStarted","Data":"77f71a014ab8516f9c91c697b0b6aff36ba2966fe6acdc7cc46b3e3c78b4612d"} Nov 01 23:00:16.252877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.251519 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b061025-9230-460b-9d8d-0ab15676b77c" containerID="77f71a014ab8516f9c91c697b0b6aff36ba2966fe6acdc7cc46b3e3c78b4612d" exitCode=60 Nov 01 23:00:16.252877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.251679 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerDied","Data":"77f71a014ab8516f9c91c697b0b6aff36ba2966fe6acdc7cc46b3e3c78b4612d"} Nov 01 23:00:16.252877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.251750 5778 scope.go:122] "RemoveContainer" containerID="5d21fce72cedb50280108ab52f389c830c6e4b61984b9d39ff5de52973633bf7" Nov 01 23:00:16.252877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.252325 5778 scope.go:122] "RemoveContainer" containerID="77f71a014ab8516f9c91c697b0b6aff36ba2966fe6acdc7cc46b3e3c78b4612d" Nov 01 23:00:16.252877 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:16.252806 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd_odh-ai-gateway-infra(4b061025-9230-460b-9d8d-0ab15676b77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" Nov 01 23:00:16.326899 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.319214 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:16.381887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.381490 5778 scope.go:122] "RemoveContainer" containerID="93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6" Nov 01 23:00:16.385876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.382656 5778 scope.go:122] "RemoveContainer" containerID="5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884" Nov 01 23:00:16.385876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:16.383490 5778 scope.go:122] "RemoveContainer" containerID="f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6" Nov 01 23:00:17.261917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.261884 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerStarted","Data":"6979eb90280646944ff9a36231bac798726fe0e73f78bd05162bd72d9ffa9ed4"} Nov 01 23:00:17.265740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.265710 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerStarted","Data":"14236dd81c64666b8e52b285bf1f67777300f7cf32c42b9c76dc4067a30d2362"} Nov 01 23:00:17.267969 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.267949 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerStarted","Data":"624f6e5de2bb68a8968ff4b4274e20ac8cc58cf12c6099e083db9f514d1bb7b3"} Nov 01 23:00:17.368912 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.367225 5778 scope.go:122] "RemoveContainer" containerID="89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69" Nov 01 23:00:17.508402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.508371 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:17.651746 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.651716 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\") pod \"4b061025-9230-460b-9d8d-0ab15676b77c\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " Nov 01 23:00:17.652149 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.652102 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca" pod "4b061025-9230-460b-9d8d-0ab15676b77c" (UID: "4b061025-9230-460b-9d8d-0ab15676b77c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:17.652149 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.652123 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\" (UniqueName: \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\") pod \"4b061025-9230-460b-9d8d-0ab15676b77c\" (UID: \"4b061025-9230-460b-9d8d-0ab15676b77c\") " Nov 01 23:00:17.653147 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.653119 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b061025-9230-460b-9d8d-0ab15676b77c-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:17.655088 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.655058 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k" pod "4b061025-9230-460b-9d8d-0ab15676b77c" (UID: "4b061025-9230-460b-9d8d-0ab15676b77c"). InnerVolumeSpecName "kube-api-access-z2x7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:17.754862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:17.754785 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2x7k\" (UniqueName: \"kubernetes.io/projected/4b061025-9230-460b-9d8d-0ab15676b77c-kube-api-access-z2x7k\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:18.279440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.279354 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerStarted","Data":"60d41a3c233bb5d9cc100038f195925849e7ad76d2e68c01d7617120d12bd9e1"} Nov 01 23:00:18.281846 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.281818 5778 generic.go:356] "Generic (PLEG): container finished" podID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerID="6979eb90280646944ff9a36231bac798726fe0e73f78bd05162bd72d9ffa9ed4" exitCode=60 Nov 01 23:00:18.282036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.281884 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerDied","Data":"6979eb90280646944ff9a36231bac798726fe0e73f78bd05162bd72d9ffa9ed4"} Nov 01 23:00:18.282036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.281952 5778 scope.go:122] "RemoveContainer" containerID="5645deabab303729e961c25d307d4d7349cdde7e9cc5d43b8dc5170e36602884" Nov 01 23:00:18.282376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.282347 5778 scope.go:122] "RemoveContainer" containerID="6979eb90280646944ff9a36231bac798726fe0e73f78bd05162bd72d9ffa9ed4" Nov 01 23:00:18.282731 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:18.282708 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p_odh-ai-gateway-infra(f9bf8dd8-f080-49fd-983f-64a7b6dd235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" Nov 01 23:00:18.283817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.283787 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" Nov 01 23:00:18.283817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.283801 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd" event={"ID":"4b061025-9230-460b-9d8d-0ab15676b77c","Type":"ContainerDied","Data":"e477e6705e88a14b03f3a653e16f044068c7041079ab98566a9201a27111cd75"} Nov 01 23:00:18.286296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.286241 5778 generic.go:356] "Generic (PLEG): container finished" podID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerID="14236dd81c64666b8e52b285bf1f67777300f7cf32c42b9c76dc4067a30d2362" exitCode=60 Nov 01 23:00:18.286457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.286434 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerDied","Data":"14236dd81c64666b8e52b285bf1f67777300f7cf32c42b9c76dc4067a30d2362"} Nov 01 23:00:18.286891 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.286829 5778 scope.go:122] "RemoveContainer" containerID="14236dd81c64666b8e52b285bf1f67777300f7cf32c42b9c76dc4067a30d2362" Nov 01 23:00:18.287193 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:18.287169 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97_odh-ai-gateway-infra(e7a106c5-77bf-4556-9fae-288669ebf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" Nov 01 23:00:18.289040 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.289014 5778 generic.go:356] "Generic (PLEG): container finished" podID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerID="624f6e5de2bb68a8968ff4b4274e20ac8cc58cf12c6099e083db9f514d1bb7b3" exitCode=60 Nov 01 23:00:18.289161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.289059 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerDied","Data":"624f6e5de2bb68a8968ff4b4274e20ac8cc58cf12c6099e083db9f514d1bb7b3"} Nov 01 23:00:18.289392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.289377 5778 scope.go:122] "RemoveContainer" containerID="624f6e5de2bb68a8968ff4b4274e20ac8cc58cf12c6099e083db9f514d1bb7b3" Nov 01 23:00:18.289648 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:18.289629 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-npbx2_odh-ai-gateway-infra(e531e6a9-516c-46c3-98fe-4490852d7f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" Nov 01 23:00:18.301058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.301035 5778 scope.go:122] "RemoveContainer" containerID="77f71a014ab8516f9c91c697b0b6aff36ba2966fe6acdc7cc46b3e3c78b4612d" Nov 01 23:00:18.313539 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.313512 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:18.315748 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.315727 5778 scope.go:122] "RemoveContainer" containerID="93dae61c3f2107e4a7e3c2045a1fa3a6ce856ca5050d865d429a200d0039dba6" Nov 01 23:00:18.334563 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.334541 5778 scope.go:122] "RemoveContainer" containerID="f2d56a9c52cc4d06111542d17b83fcab04d113a9714e5a0a8791a45bf52c03d6" Nov 01 23:00:18.339282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.339252 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:18.360675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.360650 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:18.367161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.367143 5778 scope.go:122] "RemoveContainer" containerID="2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25" Nov 01 23:00:18.426475 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.426443 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:18.431457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:18.431432 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c6cec6-29892900-n7qpd"] Nov 01 23:00:18.808092 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:18.807981 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:00:19.375232 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.375202 5778 scope.go:122] "RemoveContainer" containerID="d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41" Nov 01 23:00:19.449670 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.431727 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:19.449670 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.447790 5778 scope.go:122] "RemoveContainer" containerID="4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177" Nov 01 23:00:19.452720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.452404 5778 generic.go:356] "Generic (PLEG): container finished" podID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerID="60d41a3c233bb5d9cc100038f195925849e7ad76d2e68c01d7617120d12bd9e1" exitCode=60 Nov 01 23:00:19.452720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.452570 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerDied","Data":"60d41a3c233bb5d9cc100038f195925849e7ad76d2e68c01d7617120d12bd9e1"} Nov 01 23:00:19.452720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.452622 5778 scope.go:122] "RemoveContainer" containerID="89a341ae536b1b3195dc38bf69805dd6f1ca255c4ecd62c606c6d2d730621a69" Nov 01 23:00:19.518793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.517282 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerStarted","Data":"bf27dd4bcaee7890e959ac8cdb13f50f4ff6f173e48c5e5b81ebfd8761246580"} Nov 01 23:00:19.945836 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.945802 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:19.983349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.969995 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\") pod \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " Nov 01 23:00:19.983349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.971205 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\" (UniqueName: \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\") pod \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\" (UID: \"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c\") " Nov 01 23:00:19.983349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:19.976449 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d" pod "8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" (UID: "8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c"). InnerVolumeSpecName "kube-api-access-zmx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.038624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.032973 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca" pod "8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" (UID: "8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.038624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.033598 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmx7d\" (UniqueName: \"kubernetes.io/projected/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-kube-api-access-zmx7d\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.038624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.033625 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.086404 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.086377 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:20.092344 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.092319 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:20.239603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.239382 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\") pod \"e531e6a9-516c-46c3-98fe-4490852d7f87\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " Nov 01 23:00:20.239603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.239431 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:20.239603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.239565 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\") pod \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " Nov 01 23:00:20.239830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.239644 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\" (UniqueName: \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\") pod \"e531e6a9-516c-46c3-98fe-4490852d7f87\" (UID: \"e531e6a9-516c-46c3-98fe-4490852d7f87\") " Nov 01 23:00:20.239830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.239708 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\" (UniqueName: \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\") pod \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\" (UID: \"f9bf8dd8-f080-49fd-983f-64a7b6dd235c\") " Nov 01 23:00:20.240215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.240183 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca" pod "f9bf8dd8-f080-49fd-983f-64a7b6dd235c" (UID: "f9bf8dd8-f080-49fd-983f-64a7b6dd235c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.240748 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.240722 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.241243 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.241207 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca" pod "e531e6a9-516c-46c3-98fe-4490852d7f87" (UID: "e531e6a9-516c-46c3-98fe-4490852d7f87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.262265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.262240 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j" pod "f9bf8dd8-f080-49fd-983f-64a7b6dd235c" (UID: "f9bf8dd8-f080-49fd-983f-64a7b6dd235c"). InnerVolumeSpecName "kube-api-access-p4k7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.264078 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.264048 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7" pod "e531e6a9-516c-46c3-98fe-4490852d7f87" (UID: "e531e6a9-516c-46c3-98fe-4490852d7f87"). InnerVolumeSpecName "kube-api-access-pbln7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.341910 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.341869 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\" (UniqueName: \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\") pod \"e7a106c5-77bf-4556-9fae-288669ebf91b\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " Nov 01 23:00:20.342086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.341934 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\") pod \"e7a106c5-77bf-4556-9fae-288669ebf91b\" (UID: \"e7a106c5-77bf-4556-9fae-288669ebf91b\") " Nov 01 23:00:20.343457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.342820 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca" pod "e7a106c5-77bf-4556-9fae-288669ebf91b" (UID: "e7a106c5-77bf-4556-9fae-288669ebf91b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.343457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.342902 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e531e6a9-516c-46c3-98fe-4490852d7f87-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.343457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.342927 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbln7\" (UniqueName: \"kubernetes.io/projected/e531e6a9-516c-46c3-98fe-4490852d7f87-kube-api-access-pbln7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.343457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.342944 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k7j\" (UniqueName: \"kubernetes.io/projected/f9bf8dd8-f080-49fd-983f-64a7b6dd235c-kube-api-access-p4k7j\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.344943 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.344922 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll" pod "e7a106c5-77bf-4556-9fae-288669ebf91b" (UID: "e7a106c5-77bf-4556-9fae-288669ebf91b"). InnerVolumeSpecName "kube-api-access-qccll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.382905 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.382868 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" path="/var/lib/kubelet/pods/4b061025-9230-460b-9d8d-0ab15676b77c/volumes" Nov 01 23:00:20.445611 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.445580 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qccll\" (UniqueName: \"kubernetes.io/projected/e7a106c5-77bf-4556-9fae-288669ebf91b-kube-api-access-qccll\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.445845 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.445830 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a106c5-77bf-4556-9fae-288669ebf91b-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.567308 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.567230 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:20.581364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.581327 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerStarted","Data":"1b42522b017698d0751125ac1c96f8e70ea92b769e22dbc3b3bed745f42cdb0b"} Nov 01 23:00:20.583207 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.583182 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" Nov 01 23:00:20.583339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.583228 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v" event={"ID":"8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c","Type":"ContainerDied","Data":"fd1436f6dc2e42969c55e1db2152294fd754e205409975a0cc789a1844393c3b"} Nov 01 23:00:20.583339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.583277 5778 scope.go:122] "RemoveContainer" containerID="60d41a3c233bb5d9cc100038f195925849e7ad76d2e68c01d7617120d12bd9e1" Nov 01 23:00:20.590913 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.590886 5778 generic.go:356] "Generic (PLEG): container finished" podID="54209855-7b21-4953-94cd-2035c2506f6c" containerID="bf27dd4bcaee7890e959ac8cdb13f50f4ff6f173e48c5e5b81ebfd8761246580" exitCode=60 Nov 01 23:00:20.591131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.591109 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerDied","Data":"bf27dd4bcaee7890e959ac8cdb13f50f4ff6f173e48c5e5b81ebfd8761246580"} Nov 01 23:00:20.600867 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.600811 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" event={"ID":"f9bf8dd8-f080-49fd-983f-64a7b6dd235c","Type":"ContainerDied","Data":"364ab5616d3ed89d8f773f8eae655847aec49f145e3fd5e0bf965f3c2d69ee6a"} Nov 01 23:00:20.601440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.601347 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p" Nov 01 23:00:20.604661 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.604633 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerStarted","Data":"99ece382ecf17a67b2ffb24782ee655ea679ee13204b45064dc5501221b41100"} Nov 01 23:00:20.609045 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.609017 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" event={"ID":"e7a106c5-77bf-4556-9fae-288669ebf91b","Type":"ContainerDied","Data":"56753090b6e02b9faacc5bdee802045ffab22252a5791f81d2939b32762c7545"} Nov 01 23:00:20.609164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.609147 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97" Nov 01 23:00:20.609830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.609804 5778 scope.go:122] "RemoveContainer" containerID="2fb8398896eb84a69548161d3b174820070ad08f5894cb8c7106de40766dad25" Nov 01 23:00:20.614660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.613743 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" event={"ID":"e531e6a9-516c-46c3-98fe-4490852d7f87","Type":"ContainerDied","Data":"792b2744d94f6bf7e9df81c6828f5d2a662d669ad4ac430be58d18fe6b23cf7e"} Nov 01 23:00:20.614660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.613875 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2" Nov 01 23:00:20.635520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.635405 5778 scope.go:122] "RemoveContainer" containerID="6979eb90280646944ff9a36231bac798726fe0e73f78bd05162bd72d9ffa9ed4" Nov 01 23:00:20.638205 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.638174 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:20.646073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.646044 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892900-dgq7v"] Nov 01 23:00:20.656110 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.655644 5778 scope.go:122] "RemoveContainer" containerID="14236dd81c64666b8e52b285bf1f67777300f7cf32c42b9c76dc4067a30d2362" Nov 01 23:00:20.677292 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.677198 5778 scope.go:122] "RemoveContainer" containerID="624f6e5de2bb68a8968ff4b4274e20ac8cc58cf12c6099e083db9f514d1bb7b3" Nov 01 23:00:20.720290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.720252 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:20.727620 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.727587 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-947846-29892900-5nn7p"] Nov 01 23:00:20.738557 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.738530 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:20.747332 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.747308 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-npbx2"] Nov 01 23:00:20.771721 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.771695 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:20.778071 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.778043 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892900-nxd97"] Nov 01 23:00:20.866574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.866548 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:20.956558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.956324 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\") pod \"54209855-7b21-4953-94cd-2035c2506f6c\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " Nov 01 23:00:20.956558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.956412 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\") pod \"54209855-7b21-4953-94cd-2035c2506f6c\" (UID: \"54209855-7b21-4953-94cd-2035c2506f6c\") " Nov 01 23:00:20.958502 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.957675 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca" pod "54209855-7b21-4953-94cd-2035c2506f6c" (UID: "54209855-7b21-4953-94cd-2035c2506f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.959524 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.959483 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54209855-7b21-4953-94cd-2035c2506f6c-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.967002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.966979 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 23:00:20.988533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:20.988474 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn" pod "54209855-7b21-4953-94cd-2035c2506f6c" (UID: "54209855-7b21-4953-94cd-2035c2506f6c"). InnerVolumeSpecName "kube-api-access-9dqmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:21.134345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.099492 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/54209855-7b21-4953-94cd-2035c2506f6c-kube-api-access-9dqmn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:21.629387 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.629284 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:21.633513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.633480 5778 generic.go:356] "Generic (PLEG): container finished" podID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerID="1b42522b017698d0751125ac1c96f8e70ea92b769e22dbc3b3bed745f42cdb0b" exitCode=60 Nov 01 23:00:21.633685 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.633578 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerDied","Data":"1b42522b017698d0751125ac1c96f8e70ea92b769e22dbc3b3bed745f42cdb0b"} Nov 01 23:00:21.634604 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.634580 5778 scope.go:122] "RemoveContainer" containerID="4ceb90bf362b6b5d7bc6effcd8dab874771f1d1a0c8f0d5f24634e8a6ba6e177" Nov 01 23:00:21.639311 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.639275 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" event={"ID":"54209855-7b21-4953-94cd-2035c2506f6c","Type":"ContainerDied","Data":"a1ae22828a9b7e1843db77ffdc754850828b091b4e0e1e889ab4d1ae610cb604"} Nov 01 23:00:21.639427 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.639364 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h" Nov 01 23:00:21.645054 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.645027 5778 generic.go:356] "Generic (PLEG): container finished" podID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerID="99ece382ecf17a67b2ffb24782ee655ea679ee13204b45064dc5501221b41100" exitCode=60 Nov 01 23:00:21.645247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.645115 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerDied","Data":"99ece382ecf17a67b2ffb24782ee655ea679ee13204b45064dc5501221b41100"} Nov 01 23:00:21.645616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.645582 5778 scope.go:122] "RemoveContainer" containerID="99ece382ecf17a67b2ffb24782ee655ea679ee13204b45064dc5501221b41100" Nov 01 23:00:21.645982 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:21.645958 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p_odh-ai-gateway-infra(5df4aab5-481c-4dce-a81e-08b5e79a0043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" Nov 01 23:00:21.658471 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.658449 5778 scope.go:122] "RemoveContainer" containerID="bf27dd4bcaee7890e959ac8cdb13f50f4ff6f173e48c5e5b81ebfd8761246580" Nov 01 23:00:21.671580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.671556 5778 scope.go:122] "RemoveContainer" containerID="d6e3d0a4ce6a28f4481d2adcd1276c7a93513a88e01131be9436c363cdcd9a41" Nov 01 23:00:21.695945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.695910 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:21.706877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.704080 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:21.710099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.710073 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56b833-29892900-7544h"] Nov 01 23:00:21.858571 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.858542 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:21.959576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.959539 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\") pod \"4b4e7a4d-f6e8-430f-a48e-445b376201df\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " Nov 01 23:00:21.959795 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.959727 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\" (UniqueName: \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\") pod \"4b4e7a4d-f6e8-430f-a48e-445b376201df\" (UID: \"4b4e7a4d-f6e8-430f-a48e-445b376201df\") " Nov 01 23:00:21.960115 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.960077 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca" pod "4b4e7a4d-f6e8-430f-a48e-445b376201df" (UID: "4b4e7a4d-f6e8-430f-a48e-445b376201df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:21.960597 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.960570 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4e7a4d-f6e8-430f-a48e-445b376201df-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:21.962533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:21.962507 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv" pod "4b4e7a4d-f6e8-430f-a48e-445b376201df" (UID: "4b4e7a4d-f6e8-430f-a48e-445b376201df"). InnerVolumeSpecName "kube-api-access-sqnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:22.062125 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.062098 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnxv\" (UniqueName: \"kubernetes.io/projected/4b4e7a4d-f6e8-430f-a48e-445b376201df-kube-api-access-sqnxv\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:22.377411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.377334 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54209855-7b21-4953-94cd-2035c2506f6c" path="/var/lib/kubelet/pods/54209855-7b21-4953-94cd-2035c2506f6c/volumes" Nov 01 23:00:22.377844 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.377829 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" path="/var/lib/kubelet/pods/8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c/volumes" Nov 01 23:00:22.378326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.378310 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" path="/var/lib/kubelet/pods/e531e6a9-516c-46c3-98fe-4490852d7f87/volumes" Nov 01 23:00:22.379081 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.379062 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" path="/var/lib/kubelet/pods/e7a106c5-77bf-4556-9fae-288669ebf91b/volumes" Nov 01 23:00:22.379478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.379464 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" path="/var/lib/kubelet/pods/f9bf8dd8-f080-49fd-983f-64a7b6dd235c/volumes" Nov 01 23:00:22.665830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.665801 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" Nov 01 23:00:22.666409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.665806 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw" event={"ID":"4b4e7a4d-f6e8-430f-a48e-445b376201df","Type":"ContainerDied","Data":"d96b4a6159d7c467d8f185cc0b6b1e52cf88cd01bb0f66ff6c1aefe040682a05"} Nov 01 23:00:22.666409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.665971 5778 scope.go:122] "RemoveContainer" containerID="1b42522b017698d0751125ac1c96f8e70ea92b769e22dbc3b3bed745f42cdb0b" Nov 01 23:00:22.695026 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.694998 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:22.701684 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.701657 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892900-6jpfw"] Nov 01 23:00:22.876471 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.876445 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:22.978684 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.978590 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\") pod \"5df4aab5-481c-4dce-a81e-08b5e79a0043\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " Nov 01 23:00:22.978903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.978800 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\" (UniqueName: \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\") pod \"5df4aab5-481c-4dce-a81e-08b5e79a0043\" (UID: \"5df4aab5-481c-4dce-a81e-08b5e79a0043\") " Nov 01 23:00:22.978997 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.978960 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca" pod "5df4aab5-481c-4dce-a81e-08b5e79a0043" (UID: "5df4aab5-481c-4dce-a81e-08b5e79a0043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:22.979741 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.979717 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df4aab5-481c-4dce-a81e-08b5e79a0043-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:22.981316 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:22.981295 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz" pod "5df4aab5-481c-4dce-a81e-08b5e79a0043" (UID: "5df4aab5-481c-4dce-a81e-08b5e79a0043"). InnerVolumeSpecName "kube-api-access-js7sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:23.080975 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.080942 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7sz\" (UniqueName: \"kubernetes.io/projected/5df4aab5-481c-4dce-a81e-08b5e79a0043-kube-api-access-js7sz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.084753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.084699 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" containerID="cri-o://ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f" gracePeriod=45 Nov 01 23:00:23.398335 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.398309 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 23:00:23.499410 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499370 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\" (UniqueName: \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499618 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499515 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499618 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499573 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499618 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499607 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499838 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499776 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499838 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499810 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\") pod \"d9ccfea0-0f8e-460e-9328-15462d663175\" (UID: \"d9ccfea0-0f8e-460e-9328-15462d663175\") " Nov 01 23:00:23.499975 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499911 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:23.499975 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.499830 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:23.500161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.500143 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:23.501440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.500995 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.501440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.501125 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.501440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.501148 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.506630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.506483 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:23.507013 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.506748 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "kube-api-access-8ngvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:23.507803 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.507779 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs" pod "d9ccfea0-0f8e-460e-9328-15462d663175" (UID: "d9ccfea0-0f8e-460e-9328-15462d663175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:23.602749 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.602675 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ccfea0-0f8e-460e-9328-15462d663175-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.602749 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.602704 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngvq\" (UniqueName: \"kubernetes.io/projected/d9ccfea0-0f8e-460e-9328-15462d663175-kube-api-access-8ngvq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.602749 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.602718 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ccfea0-0f8e-460e-9328-15462d663175-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.692184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.692152 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" Nov 01 23:00:23.692753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.692160 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p" event={"ID":"5df4aab5-481c-4dce-a81e-08b5e79a0043","Type":"ContainerDied","Data":"81f6964854e3b98ac0fd86b332fa288c0c8af4f6502e1475f74dd5d5d7dd2fee"} Nov 01 23:00:23.692753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.692307 5778 scope.go:122] "RemoveContainer" containerID="99ece382ecf17a67b2ffb24782ee655ea679ee13204b45064dc5501221b41100" Nov 01 23:00:23.694653 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.694609 5778 generic.go:356] "Generic (PLEG): container finished" podID="d9ccfea0-0f8e-460e-9328-15462d663175" containerID="ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f" exitCode=0 Nov 01 23:00:23.694728 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.694704 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" event={"ID":"d9ccfea0-0f8e-460e-9328-15462d663175","Type":"ContainerDied","Data":"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f"} Nov 01 23:00:23.694797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.694730 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" event={"ID":"d9ccfea0-0f8e-460e-9328-15462d663175","Type":"ContainerDied","Data":"faa76da975b5848250a81b44b0c620c3e8b7d5d27f0f6f2ac38bf2f55207ee62"} Nov 01 23:00:23.694797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.694786 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf" Nov 01 23:00:23.710259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.710242 5778 scope.go:122] "RemoveContainer" containerID="ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f" Nov 01 23:00:23.724115 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.724054 5778 scope.go:122] "RemoveContainer" containerID="ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f" Nov 01 23:00:23.724958 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:23.724933 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f\": container with ID starting with ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f not found: ID does not exist" containerID="ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f" Nov 01 23:00:23.725098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.725074 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f"} err="failed to get container status \"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f\": rpc error: code = NotFound desc = could not find container \"ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f\": container with ID starting with ff9cf7d59a60459cf7ea6117c7c5686706601c1c09ceaaeb24f0b58867d61e0f not found: ID does not exist" Nov 01 23:00:23.737443 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.737421 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 23:00:23.745445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.745421 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-39e952/test-model-39e952-kserve-6b757bd885-jndwf"] Nov 01 23:00:23.757937 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.757915 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:23.762015 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:23.761996 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892900-k2q6p"] Nov 01 23:00:24.379081 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:24.379045 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" path="/var/lib/kubelet/pods/4b4e7a4d-f6e8-430f-a48e-445b376201df/volumes" Nov 01 23:00:24.379687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:24.379663 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" path="/var/lib/kubelet/pods/5df4aab5-481c-4dce-a81e-08b5e79a0043/volumes" Nov 01 23:00:24.380266 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:24.380244 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" path="/var/lib/kubelet/pods/d9ccfea0-0f8e-460e-9328-15462d663175/volumes" Nov 01 23:00:36.120353 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.120314 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" containerName="main" containerID="cri-o://70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64" gracePeriod=45 Nov 01 23:00:36.430259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.430229 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533013 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533063 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533129 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\" (UniqueName: \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533259 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533414 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.533874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.533483 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\") pod \"fe9edee4-af5d-467f-b2c0-340517f44925\" (UID: \"fe9edee4-af5d-467f-b2c0-340517f44925\") " Nov 01 23:00:36.534805 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.534496 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:36.534805 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.534667 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.534805 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.534668 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:36.535291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.535145 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:36.537069 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.537025 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "kube-api-access-dsn8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:36.540000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.539966 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:36.540096 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.540018 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs" pod "fe9edee4-af5d-467f-b2c0-340517f44925" (UID: "fe9edee4-af5d-467f-b2c0-340517f44925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:36.656113 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.650298 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.656113 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.650338 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.656113 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.650354 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsn8q\" (UniqueName: \"kubernetes.io/projected/fe9edee4-af5d-467f-b2c0-340517f44925-kube-api-access-dsn8q\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.656113 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.650368 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9edee4-af5d-467f-b2c0-340517f44925-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.656113 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.650382 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9edee4-af5d-467f-b2c0-340517f44925-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:36.872385 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.872352 5778 generic.go:356] "Generic (PLEG): container finished" podID="fe9edee4-af5d-467f-b2c0-340517f44925" containerID="70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64" exitCode=0 Nov 01 23:00:36.872562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.872407 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" Nov 01 23:00:36.872562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.872443 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" event={"ID":"fe9edee4-af5d-467f-b2c0-340517f44925","Type":"ContainerDied","Data":"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64"} Nov 01 23:00:36.872562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.872499 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg" event={"ID":"fe9edee4-af5d-467f-b2c0-340517f44925","Type":"ContainerDied","Data":"ed2dc679539d0777aaa7d37e5f08e1ebbe620f71019a5cf473fbd4c3c2512139"} Nov 01 23:00:36.872562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.872521 5778 scope.go:122] "RemoveContainer" containerID="70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64" Nov 01 23:00:36.889908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.889556 5778 scope.go:122] "RemoveContainer" containerID="70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64" Nov 01 23:00:36.890102 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:36.890068 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64\": container with ID starting with 70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64 not found: ID does not exist" containerID="70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64" Nov 01 23:00:36.890228 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.890111 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64"} err="failed to get container status \"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64\": rpc error: code = NotFound desc = could not find container \"70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64\": container with ID starting with 70a0831132fa777c92ce9bc38e551d06fb2f99d38fddfb0e026f85e77e3efd64 not found: ID does not exist" Nov 01 23:00:36.910677 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.910624 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 23:00:36.914527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:36.914501 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b48ab8/test-model-b48ab8-kserve-7b9555c865-f4hjg"] Nov 01 23:00:38.376329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:38.376289 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" path="/var/lib/kubelet/pods/fe9edee4-af5d-467f-b2c0-340517f44925/volumes" Nov 01 23:00:41.459874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:41.454905 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 23:00:41.459874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:41.455178 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" containerID="cri-o://2b5ba7cb2f1f1f0588af95409ab1d8c172a7b6cb1bc0af4dde0623c9ffda31c5" gracePeriod=30 Nov 01 23:00:41.732931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:41.725065 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 23:00:41.732931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:41.725359 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" containerID="cri-o://15cc74f02f1e8212d631e91b21c5de8a7cf2f0d3cdb8742e8a68ffe338d2217e" gracePeriod=30 Nov 01 23:00:42.117061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:42.116977 5778 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:00:42.117215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:42.117052 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:00:46.993875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:46.991656 5778 generic.go:356] "Generic (PLEG): container finished" podID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerID="15cc74f02f1e8212d631e91b21c5de8a7cf2f0d3cdb8742e8a68ffe338d2217e" exitCode=0 Nov 01 23:00:46.993875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:46.991765 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerDied","Data":"15cc74f02f1e8212d631e91b21c5de8a7cf2f0d3cdb8742e8a68ffe338d2217e"} Nov 01 23:00:46.993875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:46.991819 5778 scope.go:122] "RemoveContainer" containerID="6263b3139ea057744b3e4e1d5371cd59f33b68874942cf9abf3d2d266f75d191" Nov 01 23:00:46.997205 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:46.995189 5778 generic.go:356] "Generic (PLEG): container finished" podID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerID="2b5ba7cb2f1f1f0588af95409ab1d8c172a7b6cb1bc0af4dde0623c9ffda31c5" exitCode=0 Nov 01 23:00:46.997205 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:46.995391 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" event={"ID":"56f77ba7-e65c-4ac6-9478-3cd83b6567d8","Type":"ContainerDied","Data":"2b5ba7cb2f1f1f0588af95409ab1d8c172a7b6cb1bc0af4dde0623c9ffda31c5"} Nov 01 23:00:47.159395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.159359 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.214637 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.214758 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.214900 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.214938 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.214996 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.215040 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.215120 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.215211 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.215267 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\") pod \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\" (UID: \"56f77ba7-e65c-4ac6-9478-3cd83b6567d8\") " Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.217223 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.217231 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.217425 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.217481 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.217511 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.220299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.218805 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 23:00:47.234318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.234292 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.234436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.234336 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "kube-api-access-k2hc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.236039 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.236014 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.236433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.236294 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token" pod "56f77ba7-e65c-4ac6-9478-3cd83b6567d8" (UID: "56f77ba7-e65c-4ac6-9478-3cd83b6567d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.316616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.316580 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.316787 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.316672 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.316894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.316785 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.316970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.316903 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.316970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.316946 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317000 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317029 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317072 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317085 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317095 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.317143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317122 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\") pod \"3ddb88a4-e526-49c6-b065-e048812a8ab0\" (UID: \"3ddb88a4-e526-49c6-b065-e048812a8ab0\") " Nov 01 23:00:47.317459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317385 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.317620 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.317597 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.318320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.318293 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.319898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319832 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.319898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319874 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hc6\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-kube-api-access-k2hc6\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.319898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319889 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319904 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319917 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319931 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319945 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319957 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319970 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319982 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.319996 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.320009 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.320022 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f77ba7-e65c-4ac6-9478-3cd83b6567d8-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.320165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.320034 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ddb88a4-e526-49c6-b065-e048812a8ab0-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.321203 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.321162 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.321327 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.321304 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.321398 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.321344 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.321470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.321455 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2" pod "3ddb88a4-e526-49c6-b065-e048812a8ab0" (UID: "3ddb88a4-e526-49c6-b065-e048812a8ab0"). InnerVolumeSpecName "kube-api-access-gwck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.422132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.422019 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.422132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.422092 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwck2\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-kube-api-access-gwck2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.422132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.422122 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.422367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:47.422138 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ddb88a4-e526-49c6-b065-e048812a8ab0-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:48.005786 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.005752 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" Nov 01 23:00:48.005786 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.005767 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8" event={"ID":"56f77ba7-e65c-4ac6-9478-3cd83b6567d8","Type":"ContainerDied","Data":"ddc9c1ad8fcf3fe0e804e0e9f5bbfcf92990ec455efb451b5bcc6d286dae3dce"} Nov 01 23:00:48.006246 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.005840 5778 scope.go:122] "RemoveContainer" containerID="2b5ba7cb2f1f1f0588af95409ab1d8c172a7b6cb1bc0af4dde0623c9ffda31c5" Nov 01 23:00:48.008351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.008319 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" event={"ID":"3ddb88a4-e526-49c6-b065-e048812a8ab0","Type":"ContainerDied","Data":"8db02463a9ade345c56bc952bb65c3e7449455efba4129a90912afe09780f669"} Nov 01 23:00:48.008499 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.008333 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x" Nov 01 23:00:48.032839 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.032814 5778 scope.go:122] "RemoveContainer" containerID="15cc74f02f1e8212d631e91b21c5de8a7cf2f0d3cdb8742e8a68ffe338d2217e" Nov 01 23:00:48.049146 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.049111 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 23:00:48.055821 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.055783 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-2vlp8"] Nov 01 23:00:48.061693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.061637 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 23:00:48.065906 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.065882 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-sn72x"] Nov 01 23:00:48.375928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.375834 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" path="/var/lib/kubelet/pods/3ddb88a4-e526-49c6-b065-e048812a8ab0/volumes" Nov 01 23:00:48.376435 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:48.376418 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" path="/var/lib/kubelet/pods/56f77ba7-e65c-4ac6-9478-3cd83b6567d8/volumes" Nov 01 23:00:48.776258 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:48.776221 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:00:52.080518 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:52.080489 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 23:00:53.377775 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:53.377720 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 23:00:53.378330 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:53.377992 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" containerID="cri-o://d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73" gracePeriod=30 Nov 01 23:00:53.952723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:53.951479 5778 scope.go:122] "RemoveContainer" containerID="1af22d377f56239d074598613a8eb7d222b4a183203b2e5fd2ec222f07ca7011" Nov 01 23:00:53.970806 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:53.967930 5778 scope.go:122] "RemoveContainer" containerID="123fd3a27ae88bce05c174048b04aaceee210a455dac55bca74ae32939a607a7" Nov 01 23:00:53.986515 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:53.986491 5778 scope.go:122] "RemoveContainer" containerID="3597b5c3495f53162b447a173443de0538490088d4b6e56ee7fa0100d79fb1df" Nov 01 23:00:54.740299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:54.740259 5778 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:00:54.740797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:54.740320 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:00:57.081680 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.081622 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" podUID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" containerName="payload-processing" containerID="cri-o://753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e" gracePeriod=25 Nov 01 23:00:57.414661 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.414637 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 23:00:57.493086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.493051 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\") pod \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " Nov 01 23:00:57.493318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.493293 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\" (UniqueName: \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\") pod \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\" (UID: \"f342bcc0-a2df-4dbe-9ec2-18b9e144d316\") " Nov 01 23:00:57.493592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.493554 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume" pod "f342bcc0-a2df-4dbe-9ec2-18b9e144d316" (UID: "f342bcc0-a2df-4dbe-9ec2-18b9e144d316"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:57.494234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.494210 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.496597 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.496567 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr" pod "f342bcc0-a2df-4dbe-9ec2-18b9e144d316" (UID: "f342bcc0-a2df-4dbe-9ec2-18b9e144d316"). InnerVolumeSpecName "kube-api-access-9rqpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:57.595575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:57.595541 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqpr\" (UniqueName: \"kubernetes.io/projected/f342bcc0-a2df-4dbe-9ec2-18b9e144d316-kube-api-access-9rqpr\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.135077 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.135043 5778 generic.go:356] "Generic (PLEG): container finished" podID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" containerID="753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e" exitCode=0 Nov 01 23:00:58.135594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.135088 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" Nov 01 23:00:58.135594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.135131 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" event={"ID":"f342bcc0-a2df-4dbe-9ec2-18b9e144d316","Type":"ContainerDied","Data":"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e"} Nov 01 23:00:58.135594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.135190 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf" event={"ID":"f342bcc0-a2df-4dbe-9ec2-18b9e144d316","Type":"ContainerDied","Data":"44babd1d5436552175354425fe82f58d1a06b7aeebf12cbd4c8c4ed3b03d257a"} Nov 01 23:00:58.135594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.135228 5778 scope.go:122] "RemoveContainer" containerID="753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e" Nov 01 23:00:58.149182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.149163 5778 scope.go:122] "RemoveContainer" containerID="753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e" Nov 01 23:00:58.149471 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:58.149452 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e\": container with ID starting with 753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e not found: ID does not exist" containerID="753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e" Nov 01 23:00:58.149542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.149480 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e"} err="failed to get container status \"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e\": rpc error: code = NotFound desc = could not find container \"753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e\": container with ID starting with 753fd90c75beff0232e98bc53a87a263564abf0a77a1ffa8ae06c50fa9e6399e not found: ID does not exist" Nov 01 23:00:58.162239 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.162218 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 23:00:58.168603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.168578 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39e952-f75c5bb9-bq5zf"] Nov 01 23:00:58.379866 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.379815 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" path="/var/lib/kubelet/pods/f342bcc0-a2df-4dbe-9ec2-18b9e144d316/volumes" Nov 01 23:00:58.749250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.749223 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 23:00:58.935742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.935706 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.935949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.935845 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.935992 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936068 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936039 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.936112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936056 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936125 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936154 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936306 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936252 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936381 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936348 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936381 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936359 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.936474 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936424 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\") pod \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\" (UID: \"3d9ad79e-fbca-4caf-a90c-e7c832ff4647\") " Nov 01 23:00:58.936474 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936423 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:58.936692 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936671 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.936963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.936937 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.937120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.937091 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.937120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.937122 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.937312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.937138 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.937312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.937153 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.937312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.937166 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.938671 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.938646 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.938927 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.938888 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:58.939098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.939078 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "kube-api-access-sjr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:58.939418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:58.939401 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token" pod "3d9ad79e-fbca-4caf-a90c-e7c832ff4647" (UID: "3d9ad79e-fbca-4caf-a90c-e7c832ff4647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:59.039036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.038956 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.039036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.038982 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.039036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.038992 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjr2f\" (UniqueName: \"kubernetes.io/projected/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-kube-api-access-sjr2f\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.039036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.039003 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9ad79e-fbca-4caf-a90c-e7c832ff4647-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.144968 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.144933 5778 generic.go:356] "Generic (PLEG): container finished" podID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerID="d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73" exitCode=0 Nov 01 23:00:59.145414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.144982 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" Nov 01 23:00:59.145414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.145025 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" event={"ID":"3d9ad79e-fbca-4caf-a90c-e7c832ff4647","Type":"ContainerDied","Data":"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73"} Nov 01 23:00:59.145414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.145095 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd" event={"ID":"3d9ad79e-fbca-4caf-a90c-e7c832ff4647","Type":"ContainerDied","Data":"24154714ecebbaa756eb90b217b305772f4bba25d240590d8520befd5cc0401d"} Nov 01 23:00:59.145414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.145125 5778 scope.go:122] "RemoveContainer" containerID="d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73" Nov 01 23:00:59.158970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.158950 5778 scope.go:122] "RemoveContainer" containerID="d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73" Nov 01 23:00:59.159264 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:00:59.159237 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73\": container with ID starting with d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73 not found: ID does not exist" containerID="d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73" Nov 01 23:00:59.159329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.159275 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73"} err="failed to get container status \"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73\": rpc error: code = NotFound desc = could not find container \"d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73\": container with ID starting with d4541b214d522b6256073a684ca1afbf57051ea194c58a2956aa9be8b38c2b73 not found: ID does not exist" Nov 01 23:00:59.176506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.176478 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 23:00:59.182135 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:00:59.182114 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-mpbtd"] Nov 01 23:01:00.376000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:00.375968 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" path="/var/lib/kubelet/pods/3d9ad79e-fbca-4caf-a90c-e7c832ff4647/volumes" Nov 01 23:01:01.407264 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:01.407233 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 23:01:01.407720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:01.407469 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" podUID="f993c305-4768-45c5-a8db-2e7e14d13c8f" containerName="maas-api" containerID="cri-o://d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d" gracePeriod=30 Nov 01 23:01:02.094302 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.093968 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 23:01:02.175815 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.175782 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\") pod \"f993c305-4768-45c5-a8db-2e7e14d13c8f\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " Nov 01 23:01:02.176036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.175830 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\") pod \"f993c305-4768-45c5-a8db-2e7e14d13c8f\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " Nov 01 23:01:02.176036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.176002 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\") pod \"f993c305-4768-45c5-a8db-2e7e14d13c8f\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " Nov 01 23:01:02.176165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.176085 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\" (UniqueName: \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\") pod \"f993c305-4768-45c5-a8db-2e7e14d13c8f\" (UID: \"f993c305-4768-45c5-a8db-2e7e14d13c8f\") " Nov 01 23:01:02.176911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.176624 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca" pod "f993c305-4768-45c5-a8db-2e7e14d13c8f" (UID: "f993c305-4768-45c5-a8db-2e7e14d13c8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:02.176911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.176837 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f993c305-4768-45c5-a8db-2e7e14d13c8f-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:02.180832 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.180022 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4" pod "f993c305-4768-45c5-a8db-2e7e14d13c8f" (UID: "f993c305-4768-45c5-a8db-2e7e14d13c8f"). InnerVolumeSpecName "kube-api-access-s7jp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:02.183022 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.182996 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls" pod "f993c305-4768-45c5-a8db-2e7e14d13c8f" (UID: "f993c305-4768-45c5-a8db-2e7e14d13c8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:01:02.183102 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183082 5778 generic.go:356] "Generic (PLEG): container finished" podID="f993c305-4768-45c5-a8db-2e7e14d13c8f" containerID="d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d" exitCode=0 Nov 01 23:01:02.183160 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183134 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" Nov 01 23:01:02.183160 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183148 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" event={"ID":"f993c305-4768-45c5-a8db-2e7e14d13c8f","Type":"ContainerDied","Data":"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d"} Nov 01 23:01:02.183247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183231 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff" event={"ID":"f993c305-4768-45c5-a8db-2e7e14d13c8f","Type":"ContainerDied","Data":"f1b62d4d7cf5d95bc106bfb8626124205b88261ddaca9593051333a1d9d4aeed"} Nov 01 23:01:02.183291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183261 5778 scope.go:122] "RemoveContainer" containerID="d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d" Nov 01 23:01:02.183400 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.183377 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs" pod "f993c305-4768-45c5-a8db-2e7e14d13c8f" (UID: "f993c305-4768-45c5-a8db-2e7e14d13c8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:01:02.203440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.203419 5778 scope.go:122] "RemoveContainer" containerID="d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d" Nov 01 23:01:02.203724 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:02.203705 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d\": container with ID starting with d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d not found: ID does not exist" containerID="d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d" Nov 01 23:01:02.203776 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.203734 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d"} err="failed to get container status \"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d\": rpc error: code = NotFound desc = could not find container \"d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d\": container with ID starting with d797157f2afd3b689a1d90f5b9a5131e907baaf93e29e6f9c6717b068ab6721d not found: ID does not exist" Nov 01 23:01:02.278879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.278762 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:02.278879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.278794 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f993c305-4768-45c5-a8db-2e7e14d13c8f-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:02.278879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.278804 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jp4\" (UniqueName: \"kubernetes.io/projected/f993c305-4768-45c5-a8db-2e7e14d13c8f-kube-api-access-s7jp4\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:02.502677 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.502643 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 23:01:02.507370 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:02.507343 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39e952-5bfb8c874b-rqbff"] Nov 01 23:01:04.460876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:04.451765 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f993c305-4768-45c5-a8db-2e7e14d13c8f" path="/var/lib/kubelet/pods/f993c305-4768-45c5-a8db-2e7e14d13c8f/volumes" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.938151 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939196 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939217 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939233 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939241 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939255 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939265 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939277 5778 cpu_manager.go:414] "containerMap: removing container" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939285 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939299 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939306 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939315 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939322 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939332 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939339 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939349 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939358 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939375 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939382 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939403 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939410 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939421 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939427 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939437 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939444 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939462 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939470 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939481 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939487 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939509 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939516 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939532 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939539 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939558 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939565 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939578 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939585 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939599 5778 cpu_manager.go:414] "containerMap: removing container" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939606 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939619 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939626 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939642 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939648 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939668 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f993c305-4768-45c5-a8db-2e7e14d13c8f" containerName="maas-api" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939675 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f993c305-4768-45c5-a8db-2e7e14d13c8f" containerName="maas-api" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939687 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939693 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939706 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939715 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939731 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" containerName="payload-processing" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939738 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" containerName="payload-processing" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939750 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939756 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939773 5778 cpu_manager.go:414] "containerMap: removing container" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.939893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939780 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939955 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0989540-2365-40ec-a84c-1e1263072037" containerName="main" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939970 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939984 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.939992 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940001 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940009 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940021 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940030 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ccfea0-0f8e-460e-9328-15462d663175" containerName="main" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940039 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940054 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f77ba7-e65c-4ac6-9478-3cd83b6567d8" containerName="istio-proxy" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940065 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940072 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9ad79e-fbca-4caf-a90c-e7c832ff4647" containerName="istio-proxy" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940084 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f993c305-4768-45c5-a8db-2e7e14d13c8f" containerName="maas-api" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940093 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940102 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9edee4-af5d-467f-b2c0-340517f44925" containerName="main" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940111 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940122 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f342bcc0-a2df-4dbe-9ec2-18b9e144d316" containerName="payload-processing" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940130 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89e722b-3638-42dc-afdc-76afef5f3159" containerName="main" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940143 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940152 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940161 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940173 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940182 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940190 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940200 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:10.943744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.940207 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:10.965293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.946959 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:10.965293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.965188 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"default-dockercfg-c9cjm\"" Nov 01 23:01:10.965393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.965314 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"sub-test-model-ee07ad-kserve-self-signed-certs\"" Nov 01 23:01:10.965393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.965375 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"openshift-service-ca.crt\"" Nov 01 23:01:10.965486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.965466 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"kube-root-ca.crt\"" Nov 01 23:01:10.973035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:10.968438 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:01:11.077151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077111 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.077367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077294 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.077525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077502 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.077583 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077555 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.077690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077673 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qlt\" (UniqueName: \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.077728 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.077719 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.182398 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.182340 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qlt\" (UniqueName: \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.183051 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.183021 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.183783 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.183760 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.190919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.185583 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.192323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.192298 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.192553 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.192534 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.197969 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.193049 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.198117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.184224 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.198514 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.183619 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.232878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.223721 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.232878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.225160 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.248326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.245100 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qlt\" (UniqueName: \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\") pod \"sub-test-model-ee07ad-kserve-74d95f645c-62krb\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.324131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.324094 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:11.667358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:11.667329 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:01:11.669379 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:01:11.669352 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfffa1b57_d0a2_41a5_b91e_691495c31a8f.slice/crio-e61126a38328d892b473d72e0c1965f69daa9d296e307c32a0cadde8ec4b1659 WatchSource:0}: Error finding container e61126a38328d892b473d72e0c1965f69daa9d296e307c32a0cadde8ec4b1659: Status 404 returned error can't find the container with id e61126a38328d892b473d72e0c1965f69daa9d296e307c32a0cadde8ec4b1659 Nov 01 23:01:12.317900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:12.317825 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" event={"ID":"fffa1b57-d0a2-41a5-b91e-691495c31a8f","Type":"ContainerStarted","Data":"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25"} Nov 01 23:01:12.317900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:12.317897 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" event={"ID":"fffa1b57-d0a2-41a5-b91e-691495c31a8f","Type":"ContainerStarted","Data":"e61126a38328d892b473d72e0c1965f69daa9d296e307c32a0cadde8ec4b1659"} Nov 01 23:01:12.326297 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:12.326264 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:12.338080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:12.338021 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podStartSLOduration=2.338002478 podStartE2EDuration="2.338002478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:12.336632397 +0000 UTC m=+2242.112619240" watchObservedRunningTime="2026-11-01 23:01:12.338002478 +0000 UTC m=+2242.113989307" Nov 01 23:01:18.155872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.155817 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 23:01:18.156384 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.156100 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" podUID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" containerName="maas-api" containerID="cri-o://2cafecf1ef93b08f9812ee25dda2299044d65ba2a296ebd657d7b41d1c270279" gracePeriod=30 Nov 01 23:01:18.626342 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.626244 5778 generic.go:356] "Generic (PLEG): container finished" podID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" containerID="2cafecf1ef93b08f9812ee25dda2299044d65ba2a296ebd657d7b41d1c270279" exitCode=0 Nov 01 23:01:18.626742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.626712 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" event={"ID":"b35967fa-2928-4dee-a7f5-ee70e8a36b5e","Type":"ContainerDied","Data":"2cafecf1ef93b08f9812ee25dda2299044d65ba2a296ebd657d7b41d1c270279"} Nov 01 23:01:18.842897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.842007 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 23:01:18.892929 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:18.890070 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:01:18.892929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.892040 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\" (UniqueName: \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\") pod \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " Nov 01 23:01:18.892929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.892229 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\") pod \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " Nov 01 23:01:18.892929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.892325 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\") pod \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " Nov 01 23:01:18.892929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.892418 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\") pod \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\" (UID: \"b35967fa-2928-4dee-a7f5-ee70e8a36b5e\") " Nov 01 23:01:18.907885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.906317 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs" pod "b35967fa-2928-4dee-a7f5-ee70e8a36b5e" (UID: "b35967fa-2928-4dee-a7f5-ee70e8a36b5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:01:18.907885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.906823 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls" pod "b35967fa-2928-4dee-a7f5-ee70e8a36b5e" (UID: "b35967fa-2928-4dee-a7f5-ee70e8a36b5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:01:18.907885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.906832 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl" pod "b35967fa-2928-4dee-a7f5-ee70e8a36b5e" (UID: "b35967fa-2928-4dee-a7f5-ee70e8a36b5e"). InnerVolumeSpecName "kube-api-access-x5cnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:18.914978 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.914946 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca" pod "b35967fa-2928-4dee-a7f5-ee70e8a36b5e" (UID: "b35967fa-2928-4dee-a7f5-ee70e8a36b5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:18.996111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.996070 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:18.996111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.996113 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:18.996344 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.996130 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:18.996344 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:18.996144 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5cnl\" (UniqueName: \"kubernetes.io/projected/b35967fa-2928-4dee-a7f5-ee70e8a36b5e-kube-api-access-x5cnl\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:19.641003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:19.640970 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" Nov 01 23:01:19.641527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:19.641025 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj" event={"ID":"b35967fa-2928-4dee-a7f5-ee70e8a36b5e","Type":"ContainerDied","Data":"1ddb82333e0da6727b66ff85e479167cce7cd525e5d7b93e78f0b1e0ad616793"} Nov 01 23:01:19.641527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:19.641095 5778 scope.go:122] "RemoveContainer" containerID="2cafecf1ef93b08f9812ee25dda2299044d65ba2a296ebd657d7b41d1c270279" Nov 01 23:01:19.683212 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:19.683179 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 23:01:19.689043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:19.689015 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b48ab8-688cc5c88-qxkbj"] Nov 01 23:01:20.378263 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:20.378226 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" path="/var/lib/kubelet/pods/b35967fa-2928-4dee-a7f5-ee70e8a36b5e/volumes" Nov 01 23:01:21.326154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:21.326120 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:21.344844 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:21.344816 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:21.671770 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:21.671737 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:01:27.421861 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.421814 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:01:27.423167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423142 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:27.423167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423165 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423194 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423203 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423217 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423224 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423235 5778 cpu_manager.go:414] "containerMap: removing container" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423241 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54209855-7b21-4953-94cd-2035c2506f6c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423256 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423262 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423272 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423280 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423309 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" containerName="maas-api" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423315 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" containerName="maas-api" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423348 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:27.423386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423356 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423525 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df4aab5-481c-4dce-a81e-08b5e79a0043" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423545 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a106c5-77bf-4556-9fae-288669ebf91b" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423559 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35967fa-2928-4dee-a7f5-ee70e8a36b5e" containerName="maas-api" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423568 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e531e6a9-516c-46c3-98fe-4490852d7f87" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423580 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf8dd8-f080-49fd-983f-64a7b6dd235c" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423591 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0f76d4-4cdd-42eb-bfa0-8df2e7cc623c" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423601 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4e7a4d-f6e8-430f-a48e-445b376201df" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423612 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b061025-9230-460b-9d8d-0ab15676b77c" containerName="cleanup" Nov 01 23:01:27.424157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.423623 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddb88a4-e526-49c6-b065-e048812a8ab0" containerName="istio-proxy" Nov 01 23:01:27.429681 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.429555 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.437245 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.437223 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"sub-test-model-dc635e-kserve-self-signed-certs\"" Nov 01 23:01:27.437424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.437230 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:01:27.461658 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.461623 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"default-dockercfg-hxt2g\"" Nov 01 23:01:27.467804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.467779 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"kube-root-ca.crt\"" Nov 01 23:01:27.468090 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.468072 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"openshift-service-ca.crt\"" Nov 01 23:01:27.582904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.582840 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.583119 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.583039 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.583199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.583139 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.583329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.583260 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.583468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.583447 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.583619 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.583595 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927jm\" (UniqueName: \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688217 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688302 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927jm\" (UniqueName: \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688375 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688474 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688527 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.688930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.688624 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.689365 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.689227 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.696871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.690601 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.696871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.690759 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.709877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.707942 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.729871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.721294 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.754876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.751994 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927jm\" (UniqueName: \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\") pod \"sub-test-model-dc635e-kserve-668c7fc89-j757v\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:27.783879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:27.781875 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:28.182788 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:01:28.181915 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7a701a_a262_46b4_aab6_13c4c64f303b.slice/crio-a0f6270e085980b8a906bc5a5a7764c98be339bf770177acbe1b7c1934c3d518 WatchSource:0}: Error finding container a0f6270e085980b8a906bc5a5a7764c98be339bf770177acbe1b7c1934c3d518: Status 404 returned error can't find the container with id a0f6270e085980b8a906bc5a5a7764c98be339bf770177acbe1b7c1934c3d518 Nov 01 23:01:28.184808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:28.184784 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:01:28.882585 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:28.882549 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" event={"ID":"fb7a701a-a262-46b4-aab6-13c4c64f303b","Type":"ContainerStarted","Data":"7ac5b81397ca65b4590304404e8fea9d03e2ace5a2337718bf5314d20fa4385c"} Nov 01 23:01:28.883008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:28.882596 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" event={"ID":"fb7a701a-a262-46b4-aab6-13c4c64f303b","Type":"ContainerStarted","Data":"a0f6270e085980b8a906bc5a5a7764c98be339bf770177acbe1b7c1934c3d518"} Nov 01 23:01:28.902108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:28.902064 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" podStartSLOduration=1.902051664 podStartE2EDuration="1.902051664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:28.900668132 +0000 UTC m=+2258.676654967" watchObservedRunningTime="2026-11-01 23:01:28.902051664 +0000 UTC m=+2258.678038493" Nov 01 23:01:29.782361 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:29.782319 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:34.470420 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:34.470384 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 23:01:34.470829 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:34.470587 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" containerID="cri-o://d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884" gracePeriod=30 Nov 01 23:01:37.042434 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:37.042382 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7k2sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:37.042909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:37.042448 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:37.782144 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:37.782104 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:37.795326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:37.795300 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:37.987499 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:37.987468 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:01:39.796323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.796299 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 23:01:39.832464 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832429 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832506 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832554 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832629 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832722 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832760 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832796 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832834 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.832977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832842 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:39.832977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.832911 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:39.833249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833040 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:39.833249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833184 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:39.833249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833228 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:39.833804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833597 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.833804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833627 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.833804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833719 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.834066 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833816 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.834066 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.833830 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.835403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.835377 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:39.835597 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.835576 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:39.835774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.835751 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "kube-api-access-llgbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:39.934922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.934885 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\") pod \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\" (UID: \"4ff8306b-24d3-4d57-b45f-bad4ce1f906b\") " Nov 01 23:01:39.935793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.935767 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.935793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.935793 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.936032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.935809 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-kube-api-access-llgbh\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:39.937288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.937258 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo" pod "4ff8306b-24d3-4d57-b45f-bad4ce1f906b" (UID: "4ff8306b-24d3-4d57-b45f-bad4ce1f906b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:39.992403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.992367 5778 generic.go:356] "Generic (PLEG): container finished" podID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerID="d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884" exitCode=0 Nov 01 23:01:39.992556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.992412 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" Nov 01 23:01:39.992556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.992447 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerDied","Data":"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884"} Nov 01 23:01:39.992556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.992505 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf" event={"ID":"4ff8306b-24d3-4d57-b45f-bad4ce1f906b","Type":"ContainerDied","Data":"12feb6c0f2c5cae67cc5bcea79c4887fffc1afb217be7a6959b899e40f3aaafb"} Nov 01 23:01:39.992556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:39.992528 5778 scope.go:122] "RemoveContainer" containerID="d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884" Nov 01 23:01:40.012515 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.012494 5778 scope.go:122] "RemoveContainer" containerID="857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24" Nov 01 23:01:40.024083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.024064 5778 scope.go:122] "RemoveContainer" containerID="d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884" Nov 01 23:01:40.024430 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:40.024411 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884\": container with ID starting with d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884 not found: ID does not exist" containerID="d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884" Nov 01 23:01:40.024508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.024439 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884"} err="failed to get container status \"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884\": rpc error: code = NotFound desc = could not find container \"d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884\": container with ID starting with d1a2d7ae73f50b2e431534bf9d9957fde24fcb2c96ef15de115176c240806884 not found: ID does not exist" Nov 01 23:01:40.024508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.024455 5778 scope.go:122] "RemoveContainer" containerID="857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24" Nov 01 23:01:40.024739 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:40.024717 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24\": container with ID starting with 857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24 not found: ID does not exist" containerID="857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24" Nov 01 23:01:40.024808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.024750 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24"} err="failed to get container status \"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24\": rpc error: code = NotFound desc = could not find container \"857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24\": container with ID starting with 857d1c94d4d671a29374bebb2efa78266441828c102d5401ee33f55938cf4f24 not found: ID does not exist" Nov 01 23:01:40.037394 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.037365 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 23:01:40.037812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.037791 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff8306b-24d3-4d57-b45f-bad4ce1f906b-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:40.043702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.043679 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7k2sf"] Nov 01 23:01:40.377109 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:40.377039 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" path="/var/lib/kubelet/pods/4ff8306b-24d3-4d57-b45f-bad4ce1f906b/volumes" Nov 01 23:01:41.382366 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.382336 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 23:01:41.382889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.382571 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" containerID="cri-o://e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743" gracePeriod=30 Nov 01 23:01:41.387281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.387248 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 23:01:41.387474 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.387448 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" containerID="cri-o://a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe" gracePeriod=30 Nov 01 23:01:41.454809 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.454775 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 23:01:41.455071 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.455043 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" containerID="cri-o://989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f" gracePeriod=30 Nov 01 23:01:41.474295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.474261 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 23:01:41.474513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:41.474490 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" containerID="cri-o://2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381" gracePeriod=30 Nov 01 23:01:42.365641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:42.365605 5778 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:42.365973 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:42.365662 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:46.800009 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.799932 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 23:01:46.865834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.865789 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.865880 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866085 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866028 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866085 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866069 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866173 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866135 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866173 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866165 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866201 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866306 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.866373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.866357 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\") pod \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\" (UID: \"0a93e086-8ae2-44b5-998c-b8d8e4d90618\") " Nov 01 23:01:46.867304 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.867269 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.867508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.867477 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.867691 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.867669 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.874905 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.871307 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.874905 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.871893 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.897056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.897018 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.903240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.902815 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.903240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.903112 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.904295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.903598 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9" pod "0a93e086-8ae2-44b5-998c-b8d8e4d90618" (UID: "0a93e086-8ae2-44b5-998c-b8d8e4d90618"). InnerVolumeSpecName "kube-api-access-947q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.930983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.926162 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 23:01:46.930983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.928863 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 23:01:46.939352 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.939321 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 23:01:46.968393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968353 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.968582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968438 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.968582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968475 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.968582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968566 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.968762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968612 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.968762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968656 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.968762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968714 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.968762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968742 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968784 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968832 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968883 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968910 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968941 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.968978 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969027 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969091 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969153 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969205 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969278 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.969320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969307 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969351 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969385 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969443 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969488 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969529 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969549 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\") pod \"2a8bc47b-2368-4ad3-b00b-252c7118855a\" (UID: \"2a8bc47b-2368-4ad3-b00b-252c7118855a\") " Nov 01 23:01:46.969627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969601 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.970010 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969636 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\") pod \"9a0a6d38-b55b-45aa-936b-609ed3615898\" (UID: \"9a0a6d38-b55b-45aa-936b-609ed3615898\") " Nov 01 23:01:46.970010 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969668 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\") pod \"4d391df6-8e9d-428e-af18-b093f11dc5c2\" (UID: \"4d391df6-8e9d-428e-af18-b093f11dc5c2\") " Nov 01 23:01:46.970010 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.969784 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.970483 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970455 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970491 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970507 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970520 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947q9\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-kube-api-access-947q9\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970534 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970547 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970560 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970574 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970587 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970602 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a93e086-8ae2-44b5-998c-b8d8e4d90618-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970615 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0a6d38-b55b-45aa-936b-609ed3615898-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.970945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.970630 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.972026 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.971994 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.972315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.972287 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.972733 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.972705 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "kube-api-access-7796d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.976459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.976427 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.980746 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.980717 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.980942 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.980924 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.981127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.981108 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.981941 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.981843 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "kube-api-access-68zcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.982208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.982184 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.984392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.982469 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.984392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.982947 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.984392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.983185 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.984392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.983400 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.984392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.983905 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "kube-api-access-9bl5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.985012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.984990 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.987269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.985971 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.987269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.986176 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.987269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.986379 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.987689 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.987666 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.991323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.990511 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.991323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.990636 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.991323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.990981 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo" pod "4d391df6-8e9d-428e-af18-b093f11dc5c2" (UID: "4d391df6-8e9d-428e-af18-b093f11dc5c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.991565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.991423 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo" pod "9a0a6d38-b55b-45aa-936b-609ed3615898" (UID: "9a0a6d38-b55b-45aa-936b-609ed3615898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.992568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:46.992494 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token" pod "2a8bc47b-2368-4ad3-b00b-252c7118855a" (UID: "2a8bc47b-2368-4ad3-b00b-252c7118855a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072308 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-kube-api-access-68zcp\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072343 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072361 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072374 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072387 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7796d\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-kube-api-access-7796d\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072400 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072414 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d391df6-8e9d-428e-af18-b093f11dc5c2-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072427 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072441 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8bc47b-2368-4ad3-b00b-252c7118855a-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072470 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072484 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072497 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072513 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072528 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072540 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072553 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072565 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072577 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072591 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0a6d38-b55b-45aa-936b-609ed3615898-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072604 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bl5v\" (UniqueName: \"kubernetes.io/projected/4d391df6-8e9d-428e-af18-b093f11dc5c2-kube-api-access-9bl5v\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072620 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072633 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072646 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8bc47b-2368-4ad3-b00b-252c7118855a-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.072909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.072660 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d391df6-8e9d-428e-af18-b093f11dc5c2-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.077783 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.077755 5778 generic.go:356] "Generic (PLEG): container finished" podID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerID="989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f" exitCode=0 Nov 01 23:01:47.077926 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.077816 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" Nov 01 23:01:47.077926 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.077834 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" event={"ID":"4d391df6-8e9d-428e-af18-b093f11dc5c2","Type":"ContainerDied","Data":"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f"} Nov 01 23:01:47.078284 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.077960 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq" event={"ID":"4d391df6-8e9d-428e-af18-b093f11dc5c2","Type":"ContainerDied","Data":"126e982f104d2f506813b405a44c2802728bb9320a26bcfe67e43ca5e2882328"} Nov 01 23:01:47.078284 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.077983 5778 scope.go:122] "RemoveContainer" containerID="989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f" Nov 01 23:01:47.079648 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.079623 5778 generic.go:356] "Generic (PLEG): container finished" podID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerID="a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe" exitCode=0 Nov 01 23:01:47.079874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.079685 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" Nov 01 23:01:47.079874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.079714 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" event={"ID":"9a0a6d38-b55b-45aa-936b-609ed3615898","Type":"ContainerDied","Data":"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe"} Nov 01 23:01:47.079874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.079765 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7" event={"ID":"9a0a6d38-b55b-45aa-936b-609ed3615898","Type":"ContainerDied","Data":"41d944c368269514ae27e252fb9a86bdf185827360850e8e44238b4ab72f5cbc"} Nov 01 23:01:47.082188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.081593 5778 generic.go:356] "Generic (PLEG): container finished" podID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerID="e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743" exitCode=0 Nov 01 23:01:47.082188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.081643 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" Nov 01 23:01:47.082188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.081670 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" event={"ID":"0a93e086-8ae2-44b5-998c-b8d8e4d90618","Type":"ContainerDied","Data":"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743"} Nov 01 23:01:47.082188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.081713 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks" event={"ID":"0a93e086-8ae2-44b5-998c-b8d8e4d90618","Type":"ContainerDied","Data":"292a0437addef89a78ecb61f8c6623e4a083d469741b37ecff886c5708f6c035"} Nov 01 23:01:47.086201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.083643 5778 generic.go:356] "Generic (PLEG): container finished" podID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerID="2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381" exitCode=0 Nov 01 23:01:47.086201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.083698 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" event={"ID":"2a8bc47b-2368-4ad3-b00b-252c7118855a","Type":"ContainerDied","Data":"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381"} Nov 01 23:01:47.086201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.083732 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" event={"ID":"2a8bc47b-2368-4ad3-b00b-252c7118855a","Type":"ContainerDied","Data":"5755f59278838b60eb01d58b8e7b2dd2b2138085994d6823b9eadeaab7199441"} Nov 01 23:01:47.086201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.083795 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w" Nov 01 23:01:47.093942 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.093925 5778 scope.go:122] "RemoveContainer" containerID="989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f" Nov 01 23:01:47.094261 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:47.094238 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f\": container with ID starting with 989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f not found: ID does not exist" containerID="989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f" Nov 01 23:01:47.094362 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.094270 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f"} err="failed to get container status \"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f\": rpc error: code = NotFound desc = could not find container \"989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f\": container with ID starting with 989ed54c00da4f61b997c0c4a1d24a2ccbe615a3998faf213a2877a6da46100f not found: ID does not exist" Nov 01 23:01:47.094362 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.094289 5778 scope.go:122] "RemoveContainer" containerID="a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe" Nov 01 23:01:47.105648 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.105624 5778 scope.go:122] "RemoveContainer" containerID="a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe" Nov 01 23:01:47.106041 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:47.106020 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe\": container with ID starting with a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe not found: ID does not exist" containerID="a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe" Nov 01 23:01:47.106144 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.106052 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe"} err="failed to get container status \"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe\": rpc error: code = NotFound desc = could not find container \"a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe\": container with ID starting with a0313c7d989b0045ad0e5e195aac8559c6b6f1f6afca21773787a53c0b2ceafe not found: ID does not exist" Nov 01 23:01:47.106144 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.106074 5778 scope.go:122] "RemoveContainer" containerID="e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743" Nov 01 23:01:47.116562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.116538 5778 scope.go:122] "RemoveContainer" containerID="e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743" Nov 01 23:01:47.116915 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:47.116894 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743\": container with ID starting with e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743 not found: ID does not exist" containerID="e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743" Nov 01 23:01:47.116993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.116924 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743"} err="failed to get container status \"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743\": rpc error: code = NotFound desc = could not find container \"e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743\": container with ID starting with e9583f19e442079c7ad19f038c760a0827133e278021bb15357f43794e7f6743 not found: ID does not exist" Nov 01 23:01:47.116993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.116941 5778 scope.go:122] "RemoveContainer" containerID="2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381" Nov 01 23:01:47.168669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.168639 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 23:01:47.178534 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.176557 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-wflhq"] Nov 01 23:01:47.190903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.186880 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 23:01:47.190903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.190424 5778 scope.go:122] "RemoveContainer" containerID="2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381" Nov 01 23:01:47.192408 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.191593 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-7df9w"] Nov 01 23:01:47.192408 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:47.191733 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381\": container with ID starting with 2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381 not found: ID does not exist" containerID="2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381" Nov 01 23:01:47.192408 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.191770 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381"} err="failed to get container status \"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381\": rpc error: code = NotFound desc = could not find container \"2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381\": container with ID starting with 2beaaa5d4e259cd91ee109e4b6ee0e0004126804a13b8974b0bf181e85719381 not found: ID does not exist" Nov 01 23:01:47.202763 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.202732 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 23:01:47.207483 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.207453 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2dvks"] Nov 01 23:01:47.218046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.218014 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 23:01:47.225721 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:47.225692 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2rqp7"] Nov 01 23:01:48.380821 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:48.380784 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" path="/var/lib/kubelet/pods/0a93e086-8ae2-44b5-998c-b8d8e4d90618/volumes" Nov 01 23:01:48.381390 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:48.381373 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" path="/var/lib/kubelet/pods/2a8bc47b-2368-4ad3-b00b-252c7118855a/volumes" Nov 01 23:01:48.381821 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:48.381809 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" path="/var/lib/kubelet/pods/4d391df6-8e9d-428e-af18-b093f11dc5c2/volumes" Nov 01 23:01:48.382600 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:48.382583 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" path="/var/lib/kubelet/pods/9a0a6d38-b55b-45aa-936b-609ed3615898/volumes" Nov 01 23:01:48.774938 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:01:48.774901 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:01:54.145874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:54.137893 5778 scope.go:122] "RemoveContainer" containerID="9131eee833681fe590825f8178d5ac1b76d377b74121cf03d47b8d99f6ce361a" Nov 01 23:01:56.468929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:56.468894 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 23:01:56.469414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:01:56.469118 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" containerID="cri-o://b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d" gracePeriod=30 Nov 01 23:02:01.761968 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.761937 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 23:02:01.950719 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.950684 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951004 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:01.951046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951011 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951228 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951134 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951228 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951164 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951228 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951191 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951228 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951224 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951453 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951285 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951453 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951377 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:01.951453 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951432 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.951606 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.951582 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\") pod \"7ceb1464-83a9-4818-b3e6-963dcef48de5\" (UID: \"7ceb1464-83a9-4818-b3e6-963dcef48de5\") " Nov 01 23:02:01.952333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.952310 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:01.952441 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.952338 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:01.953262 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.953225 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:01.953791 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.953766 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:01.955235 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.955209 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:01.956757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.955864 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:01.957768 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.957210 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "kube-api-access-426cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:01.958757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.958708 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:02:01.958757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:01.958720 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy" pod "7ceb1464-83a9-4818-b3e6-963dcef48de5" (UID: "7ceb1464-83a9-4818-b3e6-963dcef48de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054327 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054360 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054371 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054379 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054388 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426cx\" (UniqueName: \"kubernetes.io/projected/7ceb1464-83a9-4818-b3e6-963dcef48de5-kube-api-access-426cx\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054402 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb1464-83a9-4818-b3e6-963dcef48de5-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.054421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.054414 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb1464-83a9-4818-b3e6-963dcef48de5-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:02.257650 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.257614 5778 generic.go:356] "Generic (PLEG): container finished" podID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerID="b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d" exitCode=0 Nov 01 23:02:02.257834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.257664 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" Nov 01 23:02:02.257834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.257689 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" event={"ID":"7ceb1464-83a9-4818-b3e6-963dcef48de5","Type":"ContainerDied","Data":"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d"} Nov 01 23:02:02.257834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.257755 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb" event={"ID":"7ceb1464-83a9-4818-b3e6-963dcef48de5","Type":"ContainerDied","Data":"1b4474174e140047d245e09b850c81dd74d022c4045768b0c0c8ca9fc0a327bc"} Nov 01 23:02:02.257834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.257791 5778 scope.go:122] "RemoveContainer" containerID="b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d" Nov 01 23:02:02.280011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.279979 5778 scope.go:122] "RemoveContainer" containerID="b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d" Nov 01 23:02:02.280643 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:02.280610 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d\": container with ID starting with b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d not found: ID does not exist" containerID="b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d" Nov 01 23:02:02.280757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.280655 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d"} err="failed to get container status \"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d\": rpc error: code = NotFound desc = could not find container \"b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d\": container with ID starting with b2feaf283b577c92db0a272e0a5c11143f260fa371da6e86c7e617b5ce37946d not found: ID does not exist" Nov 01 23:02:02.296867 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.296824 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 23:02:02.304517 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.304445 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-4xtpb"] Nov 01 23:02:02.377312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:02.377274 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" path="/var/lib/kubelet/pods/7ceb1464-83a9-4818-b3e6-963dcef48de5/volumes" Nov 01 23:02:18.775661 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:18.775617 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:02:24.273155 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:24.272048 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 23:02:27.553347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:27.551906 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 23:02:37.100697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:37.100655 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 23:02:39.304351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.304307 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" containerName="main" containerID="cri-o://89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42" gracePeriod=45 Nov 01 23:02:39.633776 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.633744 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 23:02:39.644110 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644088 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.644250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644143 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.644250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644233 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.644367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644270 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\" (UniqueName: \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.644422 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644401 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.644484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644379 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:39.644484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644445 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:39.644484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644477 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:39.644645 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.644511 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\") pod \"d3829ba6-8666-41c2-8028-f1d132af429d\" (UID: \"d3829ba6-8666-41c2-8028-f1d132af429d\") " Nov 01 23:02:39.645470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.645441 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:39.645595 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.645474 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:39.645595 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.645490 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:39.647807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.647779 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "kube-api-access-928sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:39.648165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.648143 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:39.648432 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.648414 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs" pod "d3829ba6-8666-41c2-8028-f1d132af429d" (UID: "d3829ba6-8666-41c2-8028-f1d132af429d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:39.673161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.673132 5778 generic.go:356] "Generic (PLEG): container finished" podID="d3829ba6-8666-41c2-8028-f1d132af429d" containerID="89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42" exitCode=0 Nov 01 23:02:39.673303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.673220 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" event={"ID":"d3829ba6-8666-41c2-8028-f1d132af429d","Type":"ContainerDied","Data":"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42"} Nov 01 23:02:39.673303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.673248 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" event={"ID":"d3829ba6-8666-41c2-8028-f1d132af429d","Type":"ContainerDied","Data":"9bb85c12be01b710b298ed329a42399dacc3e5c769ce15292bbd1e2285a3dcad"} Nov 01 23:02:39.673303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.673263 5778 scope.go:122] "RemoveContainer" containerID="89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42" Nov 01 23:02:39.673483 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.673400 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" Nov 01 23:02:39.703012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.702937 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 23:02:39.703826 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.703800 5778 scope.go:122] "RemoveContainer" containerID="89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42" Nov 01 23:02:39.704214 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:39.704192 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42\": container with ID starting with 89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42 not found: ID does not exist" containerID="89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42" Nov 01 23:02:39.704345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.704224 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42"} err="failed to get container status \"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42\": rpc error: code = NotFound desc = could not find container \"89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42\": container with ID starting with 89b3834a7185701021085d17d235b3a28b812641df52dffa5bb748a328531a42 not found: ID does not exist" Nov 01 23:02:39.712575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.712545 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89"] Nov 01 23:02:39.747214 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.747182 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3829ba6-8666-41c2-8028-f1d132af429d-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:39.747214 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.747216 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3829ba6-8666-41c2-8028-f1d132af429d-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:39.747392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:39.747231 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-928sf\" (UniqueName: \"kubernetes.io/projected/d3829ba6-8666-41c2-8028-f1d132af429d-kube-api-access-928sf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:40.378870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:40.378820 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" path="/var/lib/kubelet/pods/d3829ba6-8666-41c2-8028-f1d132af429d/volumes" Nov 01 23:02:40.583052 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:40.583008 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-c6cec6/auth-test-model-c6cec6-kserve-5d9cb497b9-nsm89" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:02:42.585817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.585777 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" podUID="56f8d433-c175-4e32-8c33-74f4cbd1603c" containerName="main" containerID="cri-o://d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca" gracePeriod=45 Nov 01 23:02:42.930737 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.930710 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 23:02:42.993723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.993693 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.993901 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.993809 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.993901 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.993873 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.994019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994002 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.994073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994058 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.994111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994072 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.994111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994085 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.994196 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994176 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\" (UniqueName: \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\") pod \"56f8d433-c175-4e32-8c33-74f4cbd1603c\" (UID: \"56f8d433-c175-4e32-8c33-74f4cbd1603c\") " Nov 01 23:02:42.994331 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994309 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.994911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994882 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.994911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994912 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.995119 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.994925 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.996687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.996662 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.997280 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.997257 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:42.997386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:42.997292 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx" pod "56f8d433-c175-4e32-8c33-74f4cbd1603c" (UID: "56f8d433-c175-4e32-8c33-74f4cbd1603c"). InnerVolumeSpecName "kube-api-access-w2skx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:43.096589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.096555 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8d433-c175-4e32-8c33-74f4cbd1603c-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:43.096589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.096587 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8d433-c175-4e32-8c33-74f4cbd1603c-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:43.096801 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.096602 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2skx\" (UniqueName: \"kubernetes.io/projected/56f8d433-c175-4e32-8c33-74f4cbd1603c-kube-api-access-w2skx\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:43.707318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.707287 5778 generic.go:356] "Generic (PLEG): container finished" podID="56f8d433-c175-4e32-8c33-74f4cbd1603c" containerID="d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca" exitCode=0 Nov 01 23:02:43.707810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.707328 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" Nov 01 23:02:43.707810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.707373 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" event={"ID":"56f8d433-c175-4e32-8c33-74f4cbd1603c","Type":"ContainerDied","Data":"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca"} Nov 01 23:02:43.707810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.707426 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m" event={"ID":"56f8d433-c175-4e32-8c33-74f4cbd1603c","Type":"ContainerDied","Data":"c3d1560f59214263df9f53b5b98380659fe6ac9ba6dd9941667c08c911df610a"} Nov 01 23:02:43.707810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.707449 5778 scope.go:122] "RemoveContainer" containerID="d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca" Nov 01 23:02:43.737383 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.737355 5778 scope.go:122] "RemoveContainer" containerID="d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca" Nov 01 23:02:43.737784 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:43.737757 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca\": container with ID starting with d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca not found: ID does not exist" containerID="d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca" Nov 01 23:02:43.737964 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.737793 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca"} err="failed to get container status \"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca\": rpc error: code = NotFound desc = could not find container \"d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca\": container with ID starting with d9bc03a5395cac531d792b19e264e4b71d1edfa20e1ea0cdcbfd5ba7dc6d4aca not found: ID does not exist" Nov 01 23:02:43.756753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.756703 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 23:02:43.762430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:43.762401 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-947846/auth-test-model-947846-kserve-6bf9b7c7f7-b5w8m"] Nov 01 23:02:44.376279 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:44.376241 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f8d433-c175-4e32-8c33-74f4cbd1603c" path="/var/lib/kubelet/pods/56f8d433-c175-4e32-8c33-74f4cbd1603c/volumes" Nov 01 23:02:48.749812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.749777 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:02:48.751247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751187 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:02:48.751247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751218 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:02:48.751247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751241 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:02:48.751247 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751249 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751265 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751272 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751297 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751305 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751320 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751327 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751339 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751348 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751380 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" containerName="main" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751388 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" containerName="main" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751404 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751411 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751431 5778 cpu_manager.go:414] "containerMap: removing container" podUID="56f8d433-c175-4e32-8c33-74f4cbd1603c" containerName="main" Nov 01 23:02:48.751581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751438 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f8d433-c175-4e32-8c33-74f4cbd1603c" containerName="main" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751621 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751640 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0a6d38-b55b-45aa-936b-609ed3615898" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751656 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceb1464-83a9-4818-b3e6-963dcef48de5" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751666 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d391df6-8e9d-428e-af18-b093f11dc5c2" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751676 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8bc47b-2368-4ad3-b00b-252c7118855a" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751691 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3829ba6-8666-41c2-8028-f1d132af429d" containerName="main" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751700 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a93e086-8ae2-44b5-998c-b8d8e4d90618" containerName="istio-proxy" Nov 01 23:02:48.752261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.751711 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f8d433-c175-4e32-8c33-74f4cbd1603c" containerName="main" Nov 01 23:02:48.763182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.763154 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.783132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.783099 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:02:48.784522 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.784495 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-337dee-serving-cert\"" Nov 01 23:02:48.806909 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:48.793994 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:02:48.890695 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.890664 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzkr2\" (UniqueName: \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.890902 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.890876 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.891051 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.891034 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.891115 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.891101 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.992570 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.992534 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.992756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.992687 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.992832 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.992768 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.992919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.992829 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzkr2\" (UniqueName: \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.993580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.993552 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.995718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.995694 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:48.995998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:48.995977 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:49.005869 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.005781 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzkr2\" (UniqueName: \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\") pod \"maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:49.091717 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.091676 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 23:02:49.098931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.098903 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:49.640603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.640397 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:02:49.646273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.646249 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:02:49.833911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:49.833756 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" event={"ID":"2d73ac8e-82bd-429b-b2b1-358d209201e9","Type":"ContainerStarted","Data":"4f77731c87ef456b4c05d5bf14c2cf85f9e4246b88c823099bdddaa7dfb3f82e"} Nov 01 23:02:50.855168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:50.855063 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" event={"ID":"2d73ac8e-82bd-429b-b2b1-358d209201e9","Type":"ContainerStarted","Data":"e9168a67319f1425d2510896bfc39458c7de264d71712db81b2f78955bca52e2"} Nov 01 23:02:50.855168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:50.855158 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:02:50.903931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:50.903839 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" podStartSLOduration=2.293953228 podStartE2EDuration="2.903822202s" totalImagesPullingTime="609.868974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:50.901465562 +0000 UTC m=+2340.677452391" watchObservedRunningTime="2026-11-01 23:02:50.903822202 +0000 UTC m=+2340.679809043" Nov 01 23:02:52.145210 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.145149 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerName="main" containerID="cri-o://b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb" gracePeriod=45 Nov 01 23:02:52.502008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.501980 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 23:02:52.588132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588100 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.588346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588235 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.588346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588257 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.588346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588302 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.588538 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588359 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\" (UniqueName: \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.588538 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.588408 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\") pod \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\" (UID: \"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0\") " Nov 01 23:02:52.589094 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.589068 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:52.589221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.589206 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:52.589336 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.589322 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:52.592541 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.592508 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "kube-api-access-4kbxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:52.592661 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.592606 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:52.599003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.598970 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs" pod "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" (UID: "c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:52.690465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690428 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.690465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690463 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.691002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690478 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kbxh\" (UniqueName: \"kubernetes.io/projected/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-kube-api-access-4kbxh\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.691002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690494 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.691002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690507 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.691002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.690519 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.882037 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.881935 5778 generic.go:356] "Generic (PLEG): container finished" podID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerID="b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb" exitCode=0 Nov 01 23:02:52.882037 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.881993 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" Nov 01 23:02:52.882281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.882029 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" event={"ID":"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0","Type":"ContainerDied","Data":"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb"} Nov 01 23:02:52.882281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.882093 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7" event={"ID":"c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0","Type":"ContainerDied","Data":"1c8c72a35e629de05815a9a8050fc03fe7c3d0ba71fc70a311be2e2f40f92be1"} Nov 01 23:02:52.882281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.882114 5778 scope.go:122] "RemoveContainer" containerID="b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb" Nov 01 23:02:52.894883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.894862 5778 scope.go:122] "RemoveContainer" containerID="b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb" Nov 01 23:02:52.895174 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:02:52.895156 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb\": container with ID starting with b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb not found: ID does not exist" containerID="b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb" Nov 01 23:02:52.895225 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.895183 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb"} err="failed to get container status \"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb\": rpc error: code = NotFound desc = could not find container \"b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb\": container with ID starting with b3bd7df721cd2fdf4e0665069b589c8cfe01809f52ebf35efbed341d33824ffb not found: ID does not exist" Nov 01 23:02:52.910775 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.910748 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 23:02:52.911765 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:52.911738 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c49240/test-model-c49240-kserve-75974c7d6c-8jlw7"] Nov 01 23:02:54.390015 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:54.389124 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" path="/var/lib/kubelet/pods/c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0/volumes" Nov 01 23:02:56.879356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:02:56.879327 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:03:00.975455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.975421 5778 generic.go:356] "Generic (PLEG): container finished" podID="3d157576-5447-4574-8903-4cc537809016" containerID="b85aceedd9d9cd5428d5e60f80115a7a2bb50e2460c0590c8d8ff620c08bb078" exitCode=0 Nov 01 23:03:00.975939 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.975502 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerDied","Data":"b85aceedd9d9cd5428d5e60f80115a7a2bb50e2460c0590c8d8ff620c08bb078"} Nov 01 23:03:00.976102 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.976082 5778 scope.go:122] "RemoveContainer" containerID="b85aceedd9d9cd5428d5e60f80115a7a2bb50e2460c0590c8d8ff620c08bb078" Nov 01 23:03:00.978268 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.978239 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:03:00.978358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.978317 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:03:00.978358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:00.978352 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:03:01.986936 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:01.986893 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerStarted","Data":"5cf43a6f7933e45cb86cc93ff35713ccb65b111e234eb5778bba4ee351b4e5c9"} Nov 01 23:03:02.358300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:02.358159 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:03:02.358566 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:02.358407 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:02.358678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:02.358598 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:03.357964 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:03.357923 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:03.358412 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:03.358006 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:04.136226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.136183 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" containerID="cri-o://96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288" gracePeriod=45 Nov 01 23:03:04.150971 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.150549 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": EOF" Nov 01 23:03:04.359379 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.358696 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:04.359379 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.358768 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:04.491697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.491667 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 23:03:04.586393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.586345 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.586645 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.586628 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.586769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.586756 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\" (UniqueName: \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.586914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.586900 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.587110 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.587096 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.587293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.587279 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\") pod \"912a564b-059a-4ee5-8abb-0b476a5480c4\" (UID: \"912a564b-059a-4ee5-8abb-0b476a5480c4\") " Nov 01 23:03:04.588332 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.588309 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:04.588634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.588618 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:04.588923 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.588904 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:04.596073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.595326 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "kube-api-access-d7vqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:04.599310 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.599283 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:04.602757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.602739 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs" pod "912a564b-059a-4ee5-8abb-0b476a5480c4" (UID: "912a564b-059a-4ee5-8abb-0b476a5480c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689595 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689628 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689638 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vqv\" (UniqueName: \"kubernetes.io/projected/912a564b-059a-4ee5-8abb-0b476a5480c4-kube-api-access-d7vqv\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689647 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912a564b-059a-4ee5-8abb-0b476a5480c4-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689656 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:04.689843 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:04.689666 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912a564b-059a-4ee5-8abb-0b476a5480c4-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:05.037283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.035524 5778 generic.go:356] "Generic (PLEG): container finished" podID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerID="96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288" exitCode=0 Nov 01 23:03:05.037283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.035777 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" event={"ID":"912a564b-059a-4ee5-8abb-0b476a5480c4","Type":"ContainerDied","Data":"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288"} Nov 01 23:03:05.037283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.036246 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" event={"ID":"912a564b-059a-4ee5-8abb-0b476a5480c4","Type":"ContainerDied","Data":"177cdfcbd2f99905f54ca0dd7b20ccf1e4901c1168b8584da49b8843405ba6da"} Nov 01 23:03:05.037283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.036293 5778 scope.go:122] "RemoveContainer" containerID="96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288" Nov 01 23:03:05.037283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.036460 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876" Nov 01 23:03:05.195928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.193244 5778 scope.go:122] "RemoveContainer" containerID="96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288" Nov 01 23:03:05.197076 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:05.196786 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288\": container with ID starting with 96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288 not found: ID does not exist" containerID="96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288" Nov 01 23:03:05.197076 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.196830 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288"} err="failed to get container status \"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288\": rpc error: code = NotFound desc = could not find container \"96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288\": container with ID starting with 96ac8c56a7ee6cf07473fe953e2a32fc1ff6c74c23c07fce9a2beb1f1571c288 not found: ID does not exist" Nov 01 23:03:05.197076 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.196992 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 23:03:05.202097 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.201872 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-64b196/test-model-64b196-kserve-65f8cb6c4c-2g876"] Nov 01 23:03:05.358319 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.358211 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:05.358319 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:05.358293 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:06.358374 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:06.358336 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:06.358810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:06.358389 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:06.376178 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:06.376147 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" path="/var/lib/kubelet/pods/912a564b-059a-4ee5-8abb-0b476a5480c4/volumes" Nov 01 23:03:07.358569 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:07.358522 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 01 23:03:07.359160 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:07.358593 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 01 23:03:08.363605 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:08.363574 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:03:08.364167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:08.363987 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:03:08.364812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:08.364788 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:03:12.161895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:12.159256 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 23:03:12.161895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:12.160198 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" containerID="cri-o://2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703" gracePeriod=30 Nov 01 23:03:17.466900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.466875 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 23:03:17.545189 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545151 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545204 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545333 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545400 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545441 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545508 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545585 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545687 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545789 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.545830 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\") pod \"e5cf342e-f2a6-4679-b075-ba699c16471e\" (UID: \"e5cf342e-f2a6-4679-b075-ba699c16471e\") " Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546096 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:17.546461 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546271 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546478 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546764 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cf342e-f2a6-4679-b075-ba699c16471e-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546784 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546798 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546811 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.547165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.546944 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:17.547980 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.547950 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "kube-api-access-vp44m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:17.548333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.548305 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:17.548767 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.548740 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:03:17.549081 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.549057 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token" pod "e5cf342e-f2a6-4679-b075-ba699c16471e" (UID: "e5cf342e-f2a6-4679-b075-ba699c16471e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:17.647744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.647708 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.647744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.647742 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp44m\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-kube-api-access-vp44m\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.647981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.647756 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.647981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.647768 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.647981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:17.647778 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cf342e-f2a6-4679-b075-ba699c16471e-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:18.223307 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.223276 5778 generic.go:356] "Generic (PLEG): container finished" podID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerID="2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703" exitCode=0 Nov 01 23:03:18.223521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.223340 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" Nov 01 23:03:18.223521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.223370 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerDied","Data":"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703"} Nov 01 23:03:18.223521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.223439 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r" event={"ID":"e5cf342e-f2a6-4679-b075-ba699c16471e","Type":"ContainerDied","Data":"95b330f16f9a2f35d77cb9df501af83f3cf903eec944c9d1f815c2b49f663532"} Nov 01 23:03:18.223521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.223469 5778 scope.go:122] "RemoveContainer" containerID="2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703" Nov 01 23:03:18.237240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.237222 5778 scope.go:122] "RemoveContainer" containerID="063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967" Nov 01 23:03:18.247606 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.247583 5778 scope.go:122] "RemoveContainer" containerID="2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703" Nov 01 23:03:18.247977 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:18.247943 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703\": container with ID starting with 2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703 not found: ID does not exist" containerID="2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703" Nov 01 23:03:18.248108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.247981 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703"} err="failed to get container status \"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703\": rpc error: code = NotFound desc = could not find container \"2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703\": container with ID starting with 2d23860fc78724b0bdce30cba116f3db61534c302404f5cfbacc84cd24ad4703 not found: ID does not exist" Nov 01 23:03:18.248108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.248008 5778 scope.go:122] "RemoveContainer" containerID="063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967" Nov 01 23:03:18.250018 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.249997 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 23:03:18.250974 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:18.250954 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967\": container with ID starting with 063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967 not found: ID does not exist" containerID="063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967" Nov 01 23:03:18.251061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.250981 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967"} err="failed to get container status \"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967\": rpc error: code = NotFound desc = could not find container \"063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967\": container with ID starting with 063ccc69efd1c5de09e866077fb4c2d8abfe2de2a086c8ae8374ea42d02ea967 not found: ID does not exist" Nov 01 23:03:18.253489 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.253470 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c49240-openshift-default-67dbc89555-df94r"] Nov 01 23:03:18.375838 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:18.375813 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" path="/var/lib/kubelet/pods/e5cf342e-f2a6-4679-b075-ba699c16471e/volumes" Nov 01 23:03:18.774469 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:18.774436 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:03:22.701600 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:22.701557 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 23:03:25.631479 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.631448 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:03:25.632154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632135 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632155 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632174 5778 cpu_manager.go:414] "containerMap: removing container" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632179 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632187 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerName="main" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632192 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerName="main" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632201 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:03:25.632226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632206 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:03:25.632458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632319 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c292b2f5-7ebf-4425-8fbd-6de7f1edb8b0" containerName="main" Nov 01 23:03:25.632458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632328 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:03:25.632458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632344 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff8306b-24d3-4d57-b45f-bad4ce1f906b" containerName="istio-proxy" Nov 01 23:03:25.632458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.632351 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="912a564b-059a-4ee5-8abb-0b476a5480c4" containerName="main" Nov 01 23:03:25.637993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.637971 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.643255 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.643228 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-337dee\"" Nov 01 23:03:25.644560 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.644539 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-337dee-dockercfg-pjdzf\"" Nov 01 23:03:25.645446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.645422 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:03:25.675273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.675244 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.675429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.675366 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj8f\" (UniqueName: \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.778741 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.778693 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj8f\" (UniqueName: \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.779086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.778995 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.818875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.779970 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.894769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.894352 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj8f\" (UniqueName: \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\") pod \"payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:25.965179 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:25.965146 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:26.256102 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:03:26.256052 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98a9dd7_b1fc_4a18_b316_78d47b0ebd87.slice/crio-612131136a74cc3ef9d9ace5f33eb3884bd1819bfa7a48076333dbb3fdd7882f WatchSource:0}: Error finding container 612131136a74cc3ef9d9ace5f33eb3884bd1819bfa7a48076333dbb3fdd7882f: Status 404 returned error can't find the container with id 612131136a74cc3ef9d9ace5f33eb3884bd1819bfa7a48076333dbb3fdd7882f Nov 01 23:03:26.258742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:26.258718 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:03:26.287745 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:26.287711 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" event={"ID":"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87","Type":"ContainerStarted","Data":"612131136a74cc3ef9d9ace5f33eb3884bd1819bfa7a48076333dbb3fdd7882f"} Nov 01 23:03:27.702669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:27.702623 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" containerID="cri-o://ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b" gracePeriod=25 Nov 01 23:03:28.333956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:28.332005 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" event={"ID":"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87","Type":"ContainerStarted","Data":"eda35e0f4fe49ba53872daa5de1fd9ac8936a22daed50fb8853c527381ab22a8"} Nov 01 23:03:28.333956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:28.332128 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:28.357410 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:28.357350 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" podStartSLOduration=1.563971999 podStartE2EDuration="3.357331748s" totalImagesPullingTime="1.793359749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:28.352084617 +0000 UTC m=+2378.128071449" watchObservedRunningTime="2026-11-01 23:03:28.357331748 +0000 UTC m=+2378.133318573" Nov 01 23:03:31.311971 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:31.311925 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:39.342029 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:39.341995 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:03:41.312174 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:41.312121 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:41.401840 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:41.401806 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 23:03:41.402337 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:41.402305 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" containerID="cri-o://8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765" gracePeriod=30 Nov 01 23:03:43.513501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:43.510976 5778 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:03:43.513501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:43.511043 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:03:46.791874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.790034 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 23:03:46.895259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895227 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895273 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895316 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895418 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895472 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895540 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895569 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:46.895803 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895619 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895803 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895685 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895803 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895763 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\") pod \"b327de33-07f0-4316-8a69-89e0211836f3\" (UID: \"b327de33-07f0-4316-8a69-89e0211836f3\") " Nov 01 23:03:46.895803 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895787 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:46.896046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.895892 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:46.896046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.896023 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:46.896272 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.896249 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:46.896653 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.896631 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327de33-07f0-4316-8a69-89e0211836f3-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.896792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.896776 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.896993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.896931 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.897219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.897203 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.897326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.897311 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.898072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.898040 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:03:46.898533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.898501 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:03:46.898677 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.898652 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:46.898770 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.898753 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9" pod "b327de33-07f0-4316-8a69-89e0211836f3" (UID: "b327de33-07f0-4316-8a69-89e0211836f3"). InnerVolumeSpecName "kube-api-access-n4qt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:46.998957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.998878 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327de33-07f0-4316-8a69-89e0211836f3-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.998957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.998904 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.998957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.998915 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327de33-07f0-4316-8a69-89e0211836f3-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:46.998957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:46.998923 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4qt9\" (UniqueName: \"kubernetes.io/projected/b327de33-07f0-4316-8a69-89e0211836f3-kube-api-access-n4qt9\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:47.527335 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.527298 5778 generic.go:356] "Generic (PLEG): container finished" podID="b327de33-07f0-4316-8a69-89e0211836f3" containerID="8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765" exitCode=0 Nov 01 23:03:47.527575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.527348 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" event={"ID":"b327de33-07f0-4316-8a69-89e0211836f3","Type":"ContainerDied","Data":"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765"} Nov 01 23:03:47.527575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.527400 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" event={"ID":"b327de33-07f0-4316-8a69-89e0211836f3","Type":"ContainerDied","Data":"68c54de8ba11b187f152cf9424e2ded75be27b3bda4dd004664b650ade936b0a"} Nov 01 23:03:47.527575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.527427 5778 scope.go:122] "RemoveContainer" containerID="8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765" Nov 01 23:03:47.527575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.527352 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75" Nov 01 23:03:47.542442 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.542194 5778 scope.go:122] "RemoveContainer" containerID="8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765" Nov 01 23:03:47.542536 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:47.542493 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765\": container with ID starting with 8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765 not found: ID does not exist" containerID="8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765" Nov 01 23:03:47.542592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.542548 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765"} err="failed to get container status \"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765\": rpc error: code = NotFound desc = could not find container \"8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765\": container with ID starting with 8e8e47200a2fb723f1d87a3d002476fcb2482570f81cd1b07d6747fdd9bd4765 not found: ID does not exist" Nov 01 23:03:47.558981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.558947 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 23:03:47.562804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:47.562781 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-vbb75"] Nov 01 23:03:48.065893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.065864 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 23:03:48.117388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.117295 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\" (UniqueName: \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\") pod \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " Nov 01 23:03:48.117591 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.117563 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\") pod \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\" (UID: \"c2db5323-de4f-433b-aefe-d5a816f5a0eb\") " Nov 01 23:03:48.118307 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.118272 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume" pod "c2db5323-de4f-433b-aefe-d5a816f5a0eb" (UID: "c2db5323-de4f-433b-aefe-d5a816f5a0eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:48.126897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.123030 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn" pod "c2db5323-de4f-433b-aefe-d5a816f5a0eb" (UID: "c2db5323-de4f-433b-aefe-d5a816f5a0eb"). InnerVolumeSpecName "kube-api-access-lx7bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:48.226876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.225719 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2db5323-de4f-433b-aefe-d5a816f5a0eb-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:48.226876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.225752 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7bn\" (UniqueName: \"kubernetes.io/projected/c2db5323-de4f-433b-aefe-d5a816f5a0eb-kube-api-access-lx7bn\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:48.382961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.382825 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b327de33-07f0-4316-8a69-89e0211836f3" path="/var/lib/kubelet/pods/b327de33-07f0-4316-8a69-89e0211836f3/volumes" Nov 01 23:03:48.538657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.538616 5778 generic.go:356] "Generic (PLEG): container finished" podID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerID="ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b" exitCode=0 Nov 01 23:03:48.538876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.538712 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" Nov 01 23:03:48.538876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.538761 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" event={"ID":"c2db5323-de4f-433b-aefe-d5a816f5a0eb","Type":"ContainerDied","Data":"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b"} Nov 01 23:03:48.538876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.538822 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf" event={"ID":"c2db5323-de4f-433b-aefe-d5a816f5a0eb","Type":"ContainerDied","Data":"23798d488f8a6e5b49ef882db02074ecb9e6c87ae6767402566fd3901dc318e8"} Nov 01 23:03:48.539093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.538890 5778 scope.go:122] "RemoveContainer" containerID="ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b" Nov 01 23:03:48.552508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.552284 5778 scope.go:122] "RemoveContainer" containerID="ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b" Nov 01 23:03:48.552657 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:48.552632 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b\": container with ID starting with ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b not found: ID does not exist" containerID="ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b" Nov 01 23:03:48.552738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.552667 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b"} err="failed to get container status \"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b\": rpc error: code = NotFound desc = could not find container \"ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b\": container with ID starting with ce54b1267e43fa634031e571c6ee8d9ebab3512721b6ffc770c29170050bc14b not found: ID does not exist" Nov 01 23:03:48.564275 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.564248 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 23:03:48.569492 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:48.569468 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-64b196-6bb8cb654c-56bxf"] Nov 01 23:03:48.774433 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:03:48.774381 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:03:50.380328 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:50.380294 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" path="/var/lib/kubelet/pods/c2db5323-de4f-433b-aefe-d5a816f5a0eb/volumes" Nov 01 23:03:50.848436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:50.848411 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:03:50.901003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:03:50.900975 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:04:04.000340 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.000303 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:04:04.001008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.000960 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" Nov 01 23:04:04.001008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.000978 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" Nov 01 23:04:04.001139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.001018 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" Nov 01 23:04:04.001139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.001023 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" Nov 01 23:04:04.001139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.001126 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2db5323-de4f-433b-aefe-d5a816f5a0eb" containerName="payload-pre-processing" Nov 01 23:04:04.001139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.001140 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b327de33-07f0-4316-8a69-89e0211836f3" containerName="istio-proxy" Nov 01 23:04:04.001364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.001148 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cf342e-f2a6-4679-b075-ba699c16471e" containerName="istio-proxy" Nov 01 23:04:04.015245 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.015205 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.027277 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.027230 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3fb5b4-serving-cert\"" Nov 01 23:04:04.032765 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.032734 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:04:04.132211 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.132165 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.132431 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.132333 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.132431 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.132395 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.132552 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.132484 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpv6k\" (UniqueName: \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.233630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.233596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.233873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.233682 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.233873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.233724 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.234007 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.233903 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpv6k\" (UniqueName: \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.234502 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.234471 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.236768 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.236741 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.237232 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.237209 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.242188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.242161 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpv6k\" (UniqueName: \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\") pod \"maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.343339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.343241 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:04.715676 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:04.715646 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:04:04.721842 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:04:04.721806 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58fae54_0661_4c07_9c3e_3e4329b24def.slice/crio-9a3d668f5ba50bf50f8ab3fa3e5c612cdbf043b2cfd357cb503ded6546bda89f WatchSource:0}: Error finding container 9a3d668f5ba50bf50f8ab3fa3e5c612cdbf043b2cfd357cb503ded6546bda89f: Status 404 returned error can't find the container with id 9a3d668f5ba50bf50f8ab3fa3e5c612cdbf043b2cfd357cb503ded6546bda89f Nov 01 23:04:05.587539 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.587452 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:04:05.592403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.592380 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.610987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.601563 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ec0da7-serving-cert\"" Nov 01 23:04:05.610987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.604565 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:04:05.675660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.675627 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.675846 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.675763 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmdj\" (UniqueName: \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.675846 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.675822 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.675846 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.675866 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.714734 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.714702 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" event={"ID":"a58fae54-0661-4c07-9c3e-3e4329b24def","Type":"ContainerStarted","Data":"7b466934cb2227de208a95dcf43aacb940f3596ced24895908aafc99096ac73b"} Nov 01 23:04:05.714934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.714750 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" event={"ID":"a58fae54-0661-4c07-9c3e-3e4329b24def","Type":"ContainerStarted","Data":"9a3d668f5ba50bf50f8ab3fa3e5c612cdbf043b2cfd357cb503ded6546bda89f"} Nov 01 23:04:05.714934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.714804 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:05.736997 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.736941 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" podStartSLOduration=2.22211228 podStartE2EDuration="2.736921364s" totalImagesPullingTime="514.809084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:05.735396373 +0000 UTC m=+2415.511383206" watchObservedRunningTime="2026-11-01 23:04:05.736921364 +0000 UTC m=+2415.512908191" Nov 01 23:04:05.777520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.777481 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.777712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.777593 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmdj\" (UniqueName: \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.777712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.777647 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.777712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.777690 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.778421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.778347 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.780240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.780216 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.780381 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.780358 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.786132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.786110 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmdj\" (UniqueName: \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\") pod \"maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:05.924516 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:05.924483 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:06.280020 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:06.279396 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:04:06.725269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:06.725235 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" event={"ID":"9cce4af7-41f1-46ef-a412-08a0203f4c5f","Type":"ContainerStarted","Data":"20bdf47375b0edeb20d780edd207a56a1b54018d4e25e350203597ff940e26ae"} Nov 01 23:04:07.738681 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:07.738640 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" event={"ID":"9cce4af7-41f1-46ef-a412-08a0203f4c5f","Type":"ContainerStarted","Data":"c191edc5be38a67a80475f378304cd1dcce55bcd20652eb7e373d0a6f6423fab"} Nov 01 23:04:07.740902 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:07.740876 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:07.763216 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:07.763154 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" podStartSLOduration=2.21945513 podStartE2EDuration="2.763135581s" totalImagesPullingTime="543.680451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:07.76009348 +0000 UTC m=+2417.536080309" watchObservedRunningTime="2026-11-01 23:04:07.763135581 +0000 UTC m=+2417.539122411" Nov 01 23:04:11.146555 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:11.146512 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 23:04:11.732922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:11.732890 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:04:14.761699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:14.761670 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:04:17.882133 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:17.882079 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 23:04:18.774831 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:04:18.774788 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:04:18.994956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:18.994904 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:04:19.000699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.000666 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:04:19.000877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.000775 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.009287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.007647 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8e3381-openshift-default-dockercfg-9v5w4\"" Nov 01 23:04:19.124485 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124382 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.124485 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124435 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.124736 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124530 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.124736 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124612 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.124736 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124639 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.124736 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124729 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.125022 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124764 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqhx\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.125022 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124809 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.125022 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.124911 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.227832 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.227963 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.227993 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228099 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228143 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqhx\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228187 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228280 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228339 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.228375 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.229357 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.229393 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.229729 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.230885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.229895 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.243083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.243053 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.271754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.271721 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.291880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.287809 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqhx\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.319105 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.293396 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.319349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.298127 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\") pod \"e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.618437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.618396 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:19.980253 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.980214 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:04:19.980889 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:04:19.980820 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28cceb0_fcd1_4831_bfb6_55af50b323b0.slice/crio-4d71f4bd34aa38927bc22f6fdd0f8625d26da2703fd9baa03e74e75562f9b9dc WatchSource:0}: Error finding container 4d71f4bd34aa38927bc22f6fdd0f8625d26da2703fd9baa03e74e75562f9b9dc: Status 404 returned error can't find the container with id 4d71f4bd34aa38927bc22f6fdd0f8625d26da2703fd9baa03e74e75562f9b9dc Nov 01 23:04:19.984749 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.984712 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:04:19.984872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.984780 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:04:19.984872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:19.984807 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:04:20.870977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:20.870915 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerStarted","Data":"b7ea7232081c585ccd4bdbcb5e08d6fd994bae7d05a3ed2e55fbc5392e0f7dd8"} Nov 01 23:04:20.871437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:20.871000 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerStarted","Data":"4d71f4bd34aa38927bc22f6fdd0f8625d26da2703fd9baa03e74e75562f9b9dc"} Nov 01 23:04:20.896581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:20.896515 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podStartSLOduration=2.8964961860000003 podStartE2EDuration="2.896496186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:20.893108332 +0000 UTC m=+2430.669095161" watchObservedRunningTime="2026-11-01 23:04:20.896496186 +0000 UTC m=+2430.672483017" Nov 01 23:04:21.619279 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:21.619239 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:21.642960 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:21.642927 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:21.880143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:21.880062 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:21.881148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:21.881118 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:04:26.300541 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.300475 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" podUID="cd116b84-4775-47c3-8981-c1c6088a6289" containerName="main" containerID="cri-o://9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614" gracePeriod=45 Nov 01 23:04:26.693879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.693105 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 23:04:26.780058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780014 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780124 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\" (UniqueName: \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780196 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780265 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780332 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780398 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\") pod \"cd116b84-4775-47c3-8981-c1c6088a6289\" (UID: \"cd116b84-4775-47c3-8981-c1c6088a6289\") " Nov 01 23:04:26.780599 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780471 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:26.780804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780726 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:26.780951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.780929 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:26.781065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.781050 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.781133 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.781067 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.781133 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.781082 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.783248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.783213 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:26.783386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.783311 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:26.783386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.783369 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r" pod "cd116b84-4775-47c3-8981-c1c6088a6289" (UID: "cd116b84-4775-47c3-8981-c1c6088a6289"). InnerVolumeSpecName "kube-api-access-dg44r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:26.882544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.882457 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd116b84-4775-47c3-8981-c1c6088a6289-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.882544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.882492 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg44r\" (UniqueName: \"kubernetes.io/projected/cd116b84-4775-47c3-8981-c1c6088a6289-kube-api-access-dg44r\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.882544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.882505 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd116b84-4775-47c3-8981-c1c6088a6289-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:26.935792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.935758 5778 generic.go:356] "Generic (PLEG): container finished" podID="cd116b84-4775-47c3-8981-c1c6088a6289" containerID="9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614" exitCode=0 Nov 01 23:04:26.935987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.935810 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" Nov 01 23:04:26.935987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.935869 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" event={"ID":"cd116b84-4775-47c3-8981-c1c6088a6289","Type":"ContainerDied","Data":"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614"} Nov 01 23:04:26.935987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.935919 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw" event={"ID":"cd116b84-4775-47c3-8981-c1c6088a6289","Type":"ContainerDied","Data":"ef193b207843309694de5238154897f5d812634c3d627f82d04dd8e1bd5b69bf"} Nov 01 23:04:26.935987 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.935942 5778 scope.go:122] "RemoveContainer" containerID="9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614" Nov 01 23:04:26.948178 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.948156 5778 scope.go:122] "RemoveContainer" containerID="9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614" Nov 01 23:04:26.948491 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:04:26.948465 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614\": container with ID starting with 9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614 not found: ID does not exist" containerID="9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614" Nov 01 23:04:26.948594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.948495 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614"} err="failed to get container status \"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614\": rpc error: code = NotFound desc = could not find container \"9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614\": container with ID starting with 9e39146cae26a5e5fe0dcf831803ccef211c645dd636c57a2287a9e2e5531614 not found: ID does not exist" Nov 01 23:04:26.961157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.961134 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 23:04:26.964414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:26.964395 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-49ab43/auth-test-model-49ab43-kserve-86d57c8df7-m8xlw"] Nov 01 23:04:28.376093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:28.376058 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd116b84-4775-47c3-8981-c1c6088a6289" path="/var/lib/kubelet/pods/cd116b84-4775-47c3-8981-c1c6088a6289/volumes" Nov 01 23:04:32.912294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:32.912252 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerName="main" containerID="cri-o://a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb" gracePeriod=45 Nov 01 23:04:32.916009 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:32.915975 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": dial tcp 10.132.0.159:8000: connect: connection refused" Nov 01 23:04:33.207296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.207271 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.332654 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.332770 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.332923 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.333047 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\" (UniqueName: \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.333106 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.333179 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\" (UniqueName: \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\") pod \"f06e25a9-0262-46e4-8305-78dbd3f67389\" (UID: \"f06e25a9-0262-46e4-8305-78dbd3f67389\") " Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.336135 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.336355 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.339344 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.341505 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:33.344032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.341701 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:33.346098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.346050 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2" pod "f06e25a9-0262-46e4-8305-78dbd3f67389" (UID: "f06e25a9-0262-46e4-8305-78dbd3f67389"). InnerVolumeSpecName "kube-api-access-56fr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:33.435048 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.434955 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.435249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.435228 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fr2\" (UniqueName: \"kubernetes.io/projected/f06e25a9-0262-46e4-8305-78dbd3f67389-kube-api-access-56fr2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.435249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.435247 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.435249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.435256 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06e25a9-0262-46e4-8305-78dbd3f67389-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.435462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.435266 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.435462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:33.435274 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06e25a9-0262-46e4-8305-78dbd3f67389-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:34.033230 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.033191 5778 generic.go:356] "Generic (PLEG): container finished" podID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerID="a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb" exitCode=0 Nov 01 23:04:34.033698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.033254 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" Nov 01 23:04:34.033698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.033270 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" event={"ID":"f06e25a9-0262-46e4-8305-78dbd3f67389","Type":"ContainerDied","Data":"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb"} Nov 01 23:04:34.033698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.033319 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd" event={"ID":"f06e25a9-0262-46e4-8305-78dbd3f67389","Type":"ContainerDied","Data":"db65559e0fe0d6575911ecd1911b416e751f0429ab27204416a571e2a1ced0aa"} Nov 01 23:04:34.033698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.033341 5778 scope.go:122] "RemoveContainer" containerID="a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb" Nov 01 23:04:34.049652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.046746 5778 scope.go:122] "RemoveContainer" containerID="a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb" Nov 01 23:04:34.050177 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:04:34.050149 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb\": container with ID starting with a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb not found: ID does not exist" containerID="a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb" Nov 01 23:04:34.050301 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.050187 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb"} err="failed to get container status \"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb\": rpc error: code = NotFound desc = could not find container \"a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb\": container with ID starting with a47e671ca8301f187c9dd689b87aebff53d568bd6be5a083a8dfdcf8aef3a0cb not found: ID does not exist" Nov 01 23:04:34.058106 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.058082 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 23:04:34.061732 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.061713 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-53dd05/auth-test-model-53dd05-kserve-59f8494757-wtlnd"] Nov 01 23:04:34.375957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:34.375874 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" path="/var/lib/kubelet/pods/f06e25a9-0262-46e4-8305-78dbd3f67389/volumes" Nov 01 23:04:48.775549 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:04:48.775494 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:04:59.474615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.474581 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:04:59.476077 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476050 5778 cpu_manager.go:414] "containerMap: removing container" podUID="cd116b84-4775-47c3-8981-c1c6088a6289" containerName="main" Nov 01 23:04:59.476077 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476075 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd116b84-4775-47c3-8981-c1c6088a6289" containerName="main" Nov 01 23:04:59.476291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476094 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerName="main" Nov 01 23:04:59.476291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476102 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerName="main" Nov 01 23:04:59.476402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476359 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd116b84-4775-47c3-8981-c1c6088a6289" containerName="main" Nov 01 23:04:59.476402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.476376 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06e25a9-0262-46e4-8305-78dbd3f67389" containerName="main" Nov 01 23:04:59.481542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.481507 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.489053 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.489025 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:04:59.491631 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.491508 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-85b1ad-openshift-default-dockercfg-2fgbd\"" Nov 01 23:04:59.604887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.601892 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.604887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.601980 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrbb\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.602095 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623015 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623060 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623154 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623265 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623310 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.631875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.623365 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.733831 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.734984 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735633 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735738 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735873 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735929 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735979 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.736005 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.736079 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrbb\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.736172 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.735544 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.736957 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.737761 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.737996 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.745061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.743153 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.759341 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.759258 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.762366 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.762318 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.769960 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.768480 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrbb\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\") pod \"e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:04:59.804695 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:04:59.802066 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:05:00.133655 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.133625 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:05:00.138546 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:00.138498 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff65332_db64_4bfe_8d8e_3076dc3f4a98.slice/crio-4e3b05af559b667cfd4dc8ba4f7684eab547f9c21cf7bb3e948d1b462626ad16 WatchSource:0}: Error finding container 4e3b05af559b667cfd4dc8ba4f7684eab547f9c21cf7bb3e948d1b462626ad16: Status 404 returned error can't find the container with id 4e3b05af559b667cfd4dc8ba4f7684eab547f9c21cf7bb3e948d1b462626ad16 Nov 01 23:05:00.141532 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.141490 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:00.141676 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.141581 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:00.141676 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.141630 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:00.479927 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.468042 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerStarted","Data":"4ef042699db460dcb3bef641267fcdde94571145a2ea38350172454bd2dbc715"} Nov 01 23:05:00.479927 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.468109 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerStarted","Data":"4e3b05af559b667cfd4dc8ba4f7684eab547f9c21cf7bb3e948d1b462626ad16"} Nov 01 23:05:00.489954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.489890 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podStartSLOduration=1.489866291 podStartE2EDuration="1.489866291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:00.471247599 +0000 UTC m=+2470.247234441" watchObservedRunningTime="2026-11-01 23:05:00.489866291 +0000 UTC m=+2470.265853121" Nov 01 23:05:00.804995 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:00.804915 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:05:01.837402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:01.835391 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:05:02.478304 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:02.478268 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:05:02.479270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:02.479246 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:05:04.256162 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.256111 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:04.282879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.280996 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:04.282879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.281046 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:04.286523 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.285130 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.298877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.289057 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.298877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.296397 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85b1ad-dockercfg-q69d5\"" Nov 01 23:05:04.298877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.296554 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85b1ad\"" Nov 01 23:05:04.298877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.298746 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:04.369439 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.369402 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.369722 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.369697 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.369935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.369913 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcj4q\" (UniqueName: \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.370187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.370167 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5l2\" (UniqueName: \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.471876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.471328 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.471876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.471392 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.471876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.471460 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcj4q\" (UniqueName: \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.471876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.471559 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5l2\" (UniqueName: \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.472965 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.472934 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.574796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.473877 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.575134 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.518941 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5l2\" (UniqueName: \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\") pod \"payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.592098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.592024 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcj4q\" (UniqueName: \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.621358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.621325 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:04.644046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.643792 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:04.920444 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:04.920411 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:04.920658 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:04.920631 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17741d3a_1ef0_4a47_81f3_1902f5f41879.slice/crio-0db705ffdbd9fcc62a5052ca342f4be678b9420459ece6445519b60afe42abd2 WatchSource:0}: Error finding container 0db705ffdbd9fcc62a5052ca342f4be678b9420459ece6445519b60afe42abd2: Status 404 returned error can't find the container with id 0db705ffdbd9fcc62a5052ca342f4be678b9420459ece6445519b60afe42abd2 Nov 01 23:05:05.006471 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.006436 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:05.577902 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.577865 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" event={"ID":"290d1cdf-8743-4e3f-8498-10adf8aa480e","Type":"ContainerStarted","Data":"f157f86bab7eef44b8e28cc58011c359bca2666b070751961525397da6547873"} Nov 01 23:05:05.578364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.577933 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:05.578364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.577957 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" event={"ID":"290d1cdf-8743-4e3f-8498-10adf8aa480e","Type":"ContainerStarted","Data":"ab2e0f2822f043f70cd1b2ce053a4293858ff02e371107c3e793c8efa1ab3e25"} Nov 01 23:05:05.579647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.579620 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" event={"ID":"17741d3a-1ef0-4a47-81f3-1902f5f41879","Type":"ContainerStarted","Data":"41e807661f50ea4049dbf76638b41c983313b168c5ff7e76903c104e409b156c"} Nov 01 23:05:05.579807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.579660 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" event={"ID":"17741d3a-1ef0-4a47-81f3-1902f5f41879","Type":"ContainerStarted","Data":"0db705ffdbd9fcc62a5052ca342f4be678b9420459ece6445519b60afe42abd2"} Nov 01 23:05:05.579807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.579697 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:05.596908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.596832 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" podStartSLOduration=1.5968212830000001 podStartE2EDuration="1.596821283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:05.593642197 +0000 UTC m=+2475.369629032" watchObservedRunningTime="2026-11-01 23:05:05.596821283 +0000 UTC m=+2475.372808136" Nov 01 23:05:05.613810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:05.613756 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" podStartSLOduration=1.613744725 podStartE2EDuration="1.613744725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:05.610362676 +0000 UTC m=+2475.386349521" watchObservedRunningTime="2026-11-01 23:05:05.613744725 +0000 UTC m=+2475.389731553" Nov 01 23:05:12.856373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:12.856333 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 23:05:16.591150 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:16.591119 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:16.591609 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:16.591525 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:18.777787 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:05:18.777728 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:05:22.120698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:22.120659 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 23:05:27.879921 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:27.879846 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" podUID="274ef331-6979-424e-a3e8-42f08379bd61" containerName="main" containerID="cri-o://dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21" gracePeriod=45 Nov 01 23:05:27.954901 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:27.954591 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" podUID="274ef331-6979-424e-a3e8-42f08379bd61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": dial tcp 10.132.0.161:8000: connect: connection refused" Nov 01 23:05:28.207267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.207235 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 23:05:28.212339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.212309 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:05:28.213088 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.213067 5778 cpu_manager.go:414] "containerMap: removing container" podUID="274ef331-6979-424e-a3e8-42f08379bd61" containerName="main" Nov 01 23:05:28.213088 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.213087 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274ef331-6979-424e-a3e8-42f08379bd61" containerName="main" Nov 01 23:05:28.213282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.213267 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="274ef331-6979-424e-a3e8-42f08379bd61" containerName="main" Nov 01 23:05:28.216911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.216887 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.228244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.228212 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c00bbf\"" Nov 01 23:05:28.228415 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.228301 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c00bbf-dockercfg-5hntq\"" Nov 01 23:05:28.260141 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.260085 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:05:28.304260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.302399 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:05:28.307370 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.307340 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.330382 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330334 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.330580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330496 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\" (UniqueName: \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.330580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330536 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\" (UniqueName: \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.330706 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330645 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.330822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330735 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:28.330822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330744 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.330941 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.330825 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\") pod \"274ef331-6979-424e-a3e8-42f08379bd61\" (UID: \"274ef331-6979-424e-a3e8-42f08379bd61\") " Nov 01 23:05:28.331069 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.331045 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:28.331364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.331335 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstfz\" (UniqueName: \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.331515 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.331499 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.331797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.331780 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.331899 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.331803 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.334310 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.334278 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:28.343347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.337112 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:28.343347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.337234 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "kube-api-access-vr5hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:28.343347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.337359 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs" pod "274ef331-6979-424e-a3e8-42f08379bd61" (UID: "274ef331-6979-424e-a3e8-42f08379bd61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:05:28.345269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.345227 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.504971 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.505412 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstfz\" (UniqueName: \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.505530 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5hc\" (UniqueName: \"kubernetes.io/projected/274ef331-6979-424e-a3e8-42f08379bd61-kube-api-access-vr5hc\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.505547 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ef331-6979-424e-a3e8-42f08379bd61-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.505562 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.505574 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ef331-6979-424e-a3e8-42f08379bd61-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:28.511931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.506738 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.607591 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.607546 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.607799 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.607675 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czpvv\" (UniqueName: \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.613776 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.613739 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstfz\" (UniqueName: \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\") pod \"payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.708804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.708763 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.709007 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.708872 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czpvv\" (UniqueName: \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.709545 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.709524 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.721019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.720987 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czpvv\" (UniqueName: \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.836437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.835109 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:28.925083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.924923 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:28.978286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.973881 5778 generic.go:356] "Generic (PLEG): container finished" podID="274ef331-6979-424e-a3e8-42f08379bd61" containerID="dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21" exitCode=0 Nov 01 23:05:28.978286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.973992 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" event={"ID":"274ef331-6979-424e-a3e8-42f08379bd61","Type":"ContainerDied","Data":"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21"} Nov 01 23:05:28.978286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.974028 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" event={"ID":"274ef331-6979-424e-a3e8-42f08379bd61","Type":"ContainerDied","Data":"c06a371a7d8c414fc88d5583a8793cb300b2f67f70a2e9c9c233d9131ad09758"} Nov 01 23:05:28.978286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.974049 5778 scope.go:122] "RemoveContainer" containerID="dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21" Nov 01 23:05:28.978286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:28.974208 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx" Nov 01 23:05:29.010973 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.009380 5778 scope.go:122] "RemoveContainer" containerID="dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21" Nov 01 23:05:29.010973 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:05:29.009697 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21\": container with ID starting with dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21 not found: ID does not exist" containerID="dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21" Nov 01 23:05:29.010973 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.009724 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21"} err="failed to get container status \"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21\": rpc error: code = NotFound desc = could not find container \"dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21\": container with ID starting with dfaf8d9ff660c45119d2881e25449729156995d689519c1c9d747aaae6dc5e21 not found: ID does not exist" Nov 01 23:05:29.029876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.029723 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 23:05:29.035242 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.035118 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-788d7b/auth-test-model-788d7b-kserve-64f569b89c-sfpmx"] Nov 01 23:05:29.173756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.173721 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:05:29.175028 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:29.174994 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74447c63_f037_42d7_92c7_62ac8a753056.slice/crio-09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be WatchSource:0}: Error finding container 09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be: Status 404 returned error can't find the container with id 09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be Nov 01 23:05:29.272343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.272318 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:05:29.273639 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:29.273609 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293c4777_ab30_42ae_afa2_c277b2a281df.slice/crio-3d745b2c9c124bbd6cb125bc485763d56dde7d727015bbf1070a29bc11ee2705 WatchSource:0}: Error finding container 3d745b2c9c124bbd6cb125bc485763d56dde7d727015bbf1070a29bc11ee2705: Status 404 returned error can't find the container with id 3d745b2c9c124bbd6cb125bc485763d56dde7d727015bbf1070a29bc11ee2705 Nov 01 23:05:29.984896 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.984819 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" event={"ID":"74447c63-f037-42d7-92c7-62ac8a753056","Type":"ContainerStarted","Data":"30ff2db5df66807e3243cf860d60a510d55e724a741df1e4c820ddce97fbab31"} Nov 01 23:05:29.984896 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.984899 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" event={"ID":"74447c63-f037-42d7-92c7-62ac8a753056","Type":"ContainerStarted","Data":"09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be"} Nov 01 23:05:29.985449 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.984931 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:29.986727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.986693 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" event={"ID":"293c4777-ab30-42ae-afa2-c277b2a281df","Type":"ContainerStarted","Data":"c176dc4748c00a30381ec308cb5d0a3341a636175b282dcb9b9ac1fda96447e2"} Nov 01 23:05:29.986889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.986756 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:29.986889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:29.986780 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" event={"ID":"293c4777-ab30-42ae-afa2-c277b2a281df","Type":"ContainerStarted","Data":"3d745b2c9c124bbd6cb125bc485763d56dde7d727015bbf1070a29bc11ee2705"} Nov 01 23:05:30.008880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:30.008819 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" podStartSLOduration=2.008805143 podStartE2EDuration="2.008805143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:30.006115332 +0000 UTC m=+2499.782102161" watchObservedRunningTime="2026-11-01 23:05:30.008805143 +0000 UTC m=+2499.784792026" Nov 01 23:05:30.029977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:30.029925 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" podStartSLOduration=2.029910777 podStartE2EDuration="2.029910777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:30.026479727 +0000 UTC m=+2499.802466561" watchObservedRunningTime="2026-11-01 23:05:30.029910777 +0000 UTC m=+2499.805897605" Nov 01 23:05:30.376416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:30.376334 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274ef331-6979-424e-a3e8-42f08379bd61" path="/var/lib/kubelet/pods/274ef331-6979-424e-a3e8-42f08379bd61/volumes" Nov 01 23:05:36.061159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.061113 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:05:36.076225 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.069571 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.077899 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.076718 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-00185d-openshift-default-dockercfg-dpkrg\"" Nov 01 23:05:36.104669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.102324 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130431 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130558 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130598 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130642 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130718 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130776 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.130833 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.137364 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.137647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.137421 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2kq\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.260889 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.260987 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261046 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261104 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261156 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261199 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2kq\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261301 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261406 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.261878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.261441 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.287127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.287090 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.287945 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.287916 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.293403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.293372 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.293880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.293834 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.301873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.299059 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.390227 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.387609 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2kq\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.390227 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.388349 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.392027 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.390885 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.403694 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.402804 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\") pod \"e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.415973 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.413184 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:36.737543 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:36.737507 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eab4d2_d13d_4d44_9331_93de4cb81141.slice/crio-5d677637902af3fe1790a7a168cba49b507a4e31e0687c4de1c5d5b29f7d13f8 WatchSource:0}: Error finding container 5d677637902af3fe1790a7a168cba49b507a4e31e0687c4de1c5d5b29f7d13f8: Status 404 returned error can't find the container with id 5d677637902af3fe1790a7a168cba49b507a4e31e0687c4de1c5d5b29f7d13f8 Nov 01 23:05:36.737865 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.737818 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:05:36.741209 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.741168 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:36.741299 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.741259 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:36.741340 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:36.741305 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:37.067093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.066984 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerStarted","Data":"bd275253900f9a5311ec3e4ea3eaea18f136f912e717c90cd114627bae5117ed"} Nov 01 23:05:37.067093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.067055 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerStarted","Data":"5d677637902af3fe1790a7a168cba49b507a4e31e0687c4de1c5d5b29f7d13f8"} Nov 01 23:05:37.091710 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.091633 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podStartSLOduration=1.091612991 podStartE2EDuration="1.091612991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:37.085458939 +0000 UTC m=+2506.861445770" watchObservedRunningTime="2026-11-01 23:05:37.091612991 +0000 UTC m=+2506.867599820" Nov 01 23:05:37.159635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.159575 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerName="main" containerID="cri-o://0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d" gracePeriod=45 Nov 01 23:05:37.286711 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.286674 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/ready\": dial tcp 10.132.0.162:8000: connect: connection refused" Nov 01 23:05:37.445940 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.445817 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:37.455479 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.455155 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:37.512585 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.512557 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 23:05:37.587473 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.587432 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.587703 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.587576 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.587829 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.587794 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:37.588057 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.588033 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\" (UniqueName: \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.588195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.588175 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.588347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.588320 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.588451 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.588435 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\") pod \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\" (UID: \"a0c90268-84cd-494d-ae67-4e3f3d7ec094\") " Nov 01 23:05:37.589270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.589246 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:37.589894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.589811 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:37.590219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.590197 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:37.591778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.591744 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:05:37.591903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.591817 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:05:37.592619 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.592601 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k" pod "a0c90268-84cd-494d-ae67-4e3f3d7ec094" (UID: "a0c90268-84cd-494d-ae67-4e3f3d7ec094"). InnerVolumeSpecName "kube-api-access-mrn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:37.691281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.691247 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:37.691281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.691278 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:37.691281 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.691287 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:37.691617 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.691297 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrn9k\" (UniqueName: \"kubernetes.io/projected/a0c90268-84cd-494d-ae67-4e3f3d7ec094-kube-api-access-mrn9k\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:37.691617 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:37.691308 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c90268-84cd-494d-ae67-4e3f3d7ec094-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:38.098559 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.096991 5778 generic.go:356] "Generic (PLEG): container finished" podID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerID="0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d" exitCode=0 Nov 01 23:05:38.102982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.099386 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" Nov 01 23:05:38.113614 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.111071 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" event={"ID":"a0c90268-84cd-494d-ae67-4e3f3d7ec094","Type":"ContainerDied","Data":"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d"} Nov 01 23:05:38.113614 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.111136 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc" event={"ID":"a0c90268-84cd-494d-ae67-4e3f3d7ec094","Type":"ContainerDied","Data":"05ea5b439e7ead8bf4f2315a7f96e1e185e9403c382110e929f0988f0a89a3b2"} Nov 01 23:05:38.113614 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.111160 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:38.113614 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.111194 5778 scope.go:122] "RemoveContainer" containerID="0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d" Nov 01 23:05:38.158261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.156081 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:05:38.183240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.183194 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 23:05:38.188956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.188387 5778 scope.go:122] "RemoveContainer" containerID="0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d" Nov 01 23:05:38.189103 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.189064 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d73b25/auth-test-model-d73b25-kserve-7c7bf5df-kzbwc"] Nov 01 23:05:38.189500 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:05:38.189324 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d\": container with ID starting with 0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d not found: ID does not exist" containerID="0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d" Nov 01 23:05:38.189500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.189360 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d"} err="failed to get container status \"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d\": rpc error: code = NotFound desc = could not find container \"0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d\": container with ID starting with 0e13ca4637b8b13cb10c66c946591739e5fda2c23e58a551615028cc5d9eca1d not found: ID does not exist" Nov 01 23:05:38.379574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:38.379471 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" path="/var/lib/kubelet/pods/a0c90268-84cd-494d-ae67-4e3f3d7ec094/volumes" Nov 01 23:05:40.576663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.576611 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:05:40.581917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.581887 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerName="main" Nov 01 23:05:40.590829 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.590800 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerName="main" Nov 01 23:05:40.591546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.591515 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c90268-84cd-494d-ae67-4e3f3d7ec094" containerName="main" Nov 01 23:05:40.598883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.598834 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:05:40.599050 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.599000 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.603708 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.603683 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-00185d\"" Nov 01 23:05:40.619463 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.619435 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-00185d-dockercfg-cw8b2\"" Nov 01 23:05:40.662831 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.662795 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xf5\" (UniqueName: \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.663062 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.662942 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.767876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.766029 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xf5\" (UniqueName: \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.767876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.766161 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.767876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.767261 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.835926 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.835593 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xf5\" (UniqueName: \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\") pod \"payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:40.934876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:40.928123 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:41.007900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:41.004196 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:05:41.007900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:41.005896 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:05:41.333915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:41.333873 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:05:41.389503 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:41.389221 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7720b68e_9062_45ee_bf34_a240f4b9c13f.slice/crio-350f42439f8b674581ae60ba9488a06f3412e58f0ba9bef510018f34c3cc1de8 WatchSource:0}: Error finding container 350f42439f8b674581ae60ba9488a06f3412e58f0ba9bef510018f34c3cc1de8: Status 404 returned error can't find the container with id 350f42439f8b674581ae60ba9488a06f3412e58f0ba9bef510018f34c3cc1de8 Nov 01 23:05:42.163904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.159068 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-cf2671"] Nov 01 23:05:42.174875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.164821 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:42.186878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.183071 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 23:05:42.195187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.194419 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-cf2671"] Nov 01 23:05:42.239889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.237645 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" event={"ID":"7720b68e-9062-45ee-bf34-a240f4b9c13f","Type":"ContainerStarted","Data":"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762"} Nov 01 23:05:42.239889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.237721 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" event={"ID":"7720b68e-9062-45ee-bf34-a240f4b9c13f","Type":"ContainerStarted","Data":"350f42439f8b674581ae60ba9488a06f3412e58f0ba9bef510018f34c3cc1de8"} Nov 01 23:05:42.239889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.237991 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:42.260255 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.260211 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx5n\" (UniqueName: \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\") pod \"test-curl-6983-cf2671\" (UID: \"b276c233-8165-4940-b8e7-d5b3fe01b67b\") " pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:42.274102 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.274056 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" podStartSLOduration=2.274042433 podStartE2EDuration="2.274042433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:42.270938783 +0000 UTC m=+2512.046925614" watchObservedRunningTime="2026-11-01 23:05:42.274042433 +0000 UTC m=+2512.050029276" Nov 01 23:05:42.362114 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.362075 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx5n\" (UniqueName: \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\") pod \"test-curl-6983-cf2671\" (UID: \"b276c233-8165-4940-b8e7-d5b3fe01b67b\") " pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:42.381407 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.381372 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx5n\" (UniqueName: \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\") pod \"test-curl-6983-cf2671\" (UID: \"b276c233-8165-4940-b8e7-d5b3fe01b67b\") " pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:42.545961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.540100 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:42.903121 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:42.899971 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-cf2671"] Nov 01 23:05:42.904132 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:42.904098 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb276c233_8165_4940_b8e7_d5b3fe01b67b.slice/crio-fce4a051254acd331921dacc5b12b00e98ac813930a445c7717b01f2016e5cb5 WatchSource:0}: Error finding container fce4a051254acd331921dacc5b12b00e98ac813930a445c7717b01f2016e5cb5: Status 404 returned error can't find the container with id fce4a051254acd331921dacc5b12b00e98ac813930a445c7717b01f2016e5cb5 Nov 01 23:05:43.254222 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:43.254170 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-cf2671" event={"ID":"b276c233-8165-4940-b8e7-d5b3fe01b67b","Type":"ContainerStarted","Data":"4d9eccd057e2d1dfcb42360f60e74c2154bc479936ad008fe6399f7b1dcb6663"} Nov 01 23:05:43.254721 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:43.254239 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-cf2671" event={"ID":"b276c233-8165-4940-b8e7-d5b3fe01b67b","Type":"ContainerStarted","Data":"fce4a051254acd331921dacc5b12b00e98ac813930a445c7717b01f2016e5cb5"} Nov 01 23:05:43.272499 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:43.272452 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-cf2671" podStartSLOduration=1.2724390890000001 podStartE2EDuration="1.272439089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:43.269288873 +0000 UTC m=+2513.045275703" watchObservedRunningTime="2026-11-01 23:05:43.272439089 +0000 UTC m=+2513.048425918" Nov 01 23:05:44.042992 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:44.042917 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-cf2671"] Nov 01 23:05:44.050116 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:44.050075 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-cf2671"] Nov 01 23:05:45.312047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:45.311986 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-cf2671" podUID="b276c233-8165-4940-b8e7-d5b3fe01b67b" containerName="test-curl-6983-cf2671" containerID="cri-o://4d9eccd057e2d1dfcb42360f60e74c2154bc479936ad008fe6399f7b1dcb6663" gracePeriod=2 Nov 01 23:05:47.379350 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.379320 5778 generic.go:356] "Generic (PLEG): container finished" podID="b276c233-8165-4940-b8e7-d5b3fe01b67b" containerID="4d9eccd057e2d1dfcb42360f60e74c2154bc479936ad008fe6399f7b1dcb6663" exitCode=137 Nov 01 23:05:47.561951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.561921 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:47.637773 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.637670 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\" (UniqueName: \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\") pod \"b276c233-8165-4940-b8e7-d5b3fe01b67b\" (UID: \"b276c233-8165-4940-b8e7-d5b3fe01b67b\") " Nov 01 23:05:47.640673 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.640638 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n" pod "b276c233-8165-4940-b8e7-d5b3fe01b67b" (UID: "b276c233-8165-4940-b8e7-d5b3fe01b67b"). InnerVolumeSpecName "kube-api-access-gxx5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:47.739686 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.739634 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx5n\" (UniqueName: \"kubernetes.io/projected/b276c233-8165-4940-b8e7-d5b3fe01b67b-kube-api-access-gxx5n\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:47.846652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.846610 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:05:47.848143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.848117 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b276c233-8165-4940-b8e7-d5b3fe01b67b" containerName="test-curl-6983-cf2671" Nov 01 23:05:47.848365 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.848348 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b276c233-8165-4940-b8e7-d5b3fe01b67b" containerName="test-curl-6983-cf2671" Nov 01 23:05:47.848644 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.848629 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b276c233-8165-4940-b8e7-d5b3fe01b67b" containerName="test-curl-6983-cf2671" Nov 01 23:05:47.871722 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.871680 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:05:47.873907 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.873158 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:47.880180 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.879779 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c00bbf-serving-cert\"" Nov 01 23:05:47.948593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.948539 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:47.948812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.948662 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:47.948812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.948693 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:47.948812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:47.948738 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdzf\" (UniqueName: \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.050074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.050037 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.050343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.050124 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.050343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.050140 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.050343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.050176 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdzf\" (UniqueName: \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.052387 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.052350 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.054215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.054187 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.057500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.057474 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.060108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.060079 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdzf\" (UniqueName: \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\") pod \"maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.202068 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.201976 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:48.231186 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.231131 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:48.261864 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.261812 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:48.379893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.379574 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b276c233-8165-4940-b8e7-d5b3fe01b67b" path="/var/lib/kubelet/pods/b276c233-8165-4940-b8e7-d5b3fe01b67b/volumes" Nov 01 23:05:48.396915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.395408 5778 scope.go:122] "RemoveContainer" containerID="4d9eccd057e2d1dfcb42360f60e74c2154bc479936ad008fe6399f7b1dcb6663" Nov 01 23:05:48.396915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.395591 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-cf2671" Nov 01 23:05:48.580920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:48.574505 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:05:48.580920 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:05:48.580410 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde1c837_949d_4349_9861_2135cda48339.slice/crio-5ba1159174dfd0e9ef18cc6f6d69e4be013da262119dd9934aed30400e8756e6 WatchSource:0}: Error finding container 5ba1159174dfd0e9ef18cc6f6d69e4be013da262119dd9934aed30400e8756e6: Status 404 returned error can't find the container with id 5ba1159174dfd0e9ef18cc6f6d69e4be013da262119dd9934aed30400e8756e6 Nov 01 23:05:48.910128 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:05:48.910084 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:05:49.405363 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:49.405323 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" event={"ID":"dde1c837-949d-4349-9861-2135cda48339","Type":"ContainerStarted","Data":"0ab8b069bfdc02a918cc85cecbc6f5589451fc559f58fc6a519ad9bc1829b1d7"} Nov 01 23:05:49.405963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:49.405379 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" event={"ID":"dde1c837-949d-4349-9861-2135cda48339","Type":"ContainerStarted","Data":"5ba1159174dfd0e9ef18cc6f6d69e4be013da262119dd9934aed30400e8756e6"} Nov 01 23:05:49.405963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:49.405411 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:51.426783 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.426741 5778 generic.go:356] "Generic (PLEG): container finished" podID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerID="b7ea7232081c585ccd4bdbcb5e08d6fd994bae7d05a3ed2e55fbc5392e0f7dd8" exitCode=0 Nov 01 23:05:51.427373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.426809 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerDied","Data":"b7ea7232081c585ccd4bdbcb5e08d6fd994bae7d05a3ed2e55fbc5392e0f7dd8"} Nov 01 23:05:51.427373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.427298 5778 scope.go:122] "RemoveContainer" containerID="b7ea7232081c585ccd4bdbcb5e08d6fd994bae7d05a3ed2e55fbc5392e0f7dd8" Nov 01 23:05:51.429413 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.429380 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:51.429581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.429470 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:51.429581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.429522 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:05:51.455584 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.455495 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" podStartSLOduration=3.914767049 podStartE2EDuration="4.455477778s" totalImagesPullingTime="540.710729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:49.459019748 +0000 UTC m=+2519.235006579" watchObservedRunningTime="2026-11-01 23:05:51.455477778 +0000 UTC m=+2521.231464608" Nov 01 23:05:51.880159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:51.880074 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:05:52.458069 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:52.458023 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerStarted","Data":"c2100bbf253eff6a116ba785acd84835f91bfb34560e2842d1821832981f00ab"} Nov 01 23:05:52.630873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:52.620910 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:05:52.630873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:52.625978 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:52.630873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:52.626041 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:53.240878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.240702 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" podUID="290d1cdf-8743-4e3f-8498-10adf8aa480e" containerName="payload-processing" containerID="cri-o://f157f86bab7eef44b8e28cc58011c359bca2666b070751961525397da6547873" gracePeriod=25 Nov 01 23:05:53.261099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.259001 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:05:53.262522 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.262488 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" podUID="17741d3a-1ef0-4a47-81f3-1902f5f41879" containerName="payload-pre-processing" containerID="cri-o://41e807661f50ea4049dbf76638b41c983313b168c5ff7e76903c104e409b156c" gracePeriod=25 Nov 01 23:05:53.504345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.504248 5778 generic.go:356] "Generic (PLEG): container finished" podID="290d1cdf-8743-4e3f-8498-10adf8aa480e" containerID="f157f86bab7eef44b8e28cc58011c359bca2666b070751961525397da6547873" exitCode=0 Nov 01 23:05:53.504994 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.504962 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" event={"ID":"290d1cdf-8743-4e3f-8498-10adf8aa480e","Type":"ContainerDied","Data":"f157f86bab7eef44b8e28cc58011c359bca2666b070751961525397da6547873"} Nov 01 23:05:53.507256 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.507238 5778 generic.go:356] "Generic (PLEG): container finished" podID="17741d3a-1ef0-4a47-81f3-1902f5f41879" containerID="41e807661f50ea4049dbf76638b41c983313b168c5ff7e76903c104e409b156c" exitCode=0 Nov 01 23:05:53.507439 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.507301 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" event={"ID":"17741d3a-1ef0-4a47-81f3-1902f5f41879","Type":"ContainerDied","Data":"41e807661f50ea4049dbf76638b41c983313b168c5ff7e76903c104e409b156c"} Nov 01 23:05:53.646880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.629051 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:53.646880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.629125 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:53.795036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.794829 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:53.882278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.882245 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:53.951889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.945885 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\" (UniqueName: \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\") pod \"290d1cdf-8743-4e3f-8498-10adf8aa480e\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " Nov 01 23:05:53.951889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.946073 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\") pod \"290d1cdf-8743-4e3f-8498-10adf8aa480e\" (UID: \"290d1cdf-8743-4e3f-8498-10adf8aa480e\") " Nov 01 23:05:53.959998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.954705 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume" pod "290d1cdf-8743-4e3f-8498-10adf8aa480e" (UID: "290d1cdf-8743-4e3f-8498-10adf8aa480e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:53.959998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:53.954891 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2" pod "290d1cdf-8743-4e3f-8498-10adf8aa480e" (UID: "290d1cdf-8743-4e3f-8498-10adf8aa480e"). InnerVolumeSpecName "kube-api-access-vq5l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:54.049261 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.049167 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\") pod \"17741d3a-1ef0-4a47-81f3-1902f5f41879\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " Nov 01 23:05:54.049637 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.049617 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\" (UniqueName: \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\") pod \"17741d3a-1ef0-4a47-81f3-1902f5f41879\" (UID: \"17741d3a-1ef0-4a47-81f3-1902f5f41879\") " Nov 01 23:05:54.050334 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.050305 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5l2\" (UniqueName: \"kubernetes.io/projected/290d1cdf-8743-4e3f-8498-10adf8aa480e-kube-api-access-vq5l2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:54.050491 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.050475 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290d1cdf-8743-4e3f-8498-10adf8aa480e-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:54.051431 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.051406 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume" pod "17741d3a-1ef0-4a47-81f3-1902f5f41879" (UID: "17741d3a-1ef0-4a47-81f3-1902f5f41879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:54.056322 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.056296 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q" pod "17741d3a-1ef0-4a47-81f3-1902f5f41879" (UID: "17741d3a-1ef0-4a47-81f3-1902f5f41879"). InnerVolumeSpecName "kube-api-access-lcj4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:54.151957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.151914 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17741d3a-1ef0-4a47-81f3-1902f5f41879-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:54.151957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.151951 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcj4q\" (UniqueName: \"kubernetes.io/projected/17741d3a-1ef0-4a47-81f3-1902f5f41879-kube-api-access-lcj4q\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:54.547471 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.547431 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" event={"ID":"290d1cdf-8743-4e3f-8498-10adf8aa480e","Type":"ContainerDied","Data":"ab2e0f2822f043f70cd1b2ce053a4293858ff02e371107c3e793c8efa1ab3e25"} Nov 01 23:05:54.548004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.547506 5778 scope.go:122] "RemoveContainer" containerID="f157f86bab7eef44b8e28cc58011c359bca2666b070751961525397da6547873" Nov 01 23:05:54.548004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.547675 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv" Nov 01 23:05:54.551184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.551153 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" Nov 01 23:05:54.551742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.551718 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc" event={"ID":"17741d3a-1ef0-4a47-81f3-1902f5f41879","Type":"ContainerDied","Data":"0db705ffdbd9fcc62a5052ca342f4be678b9420459ece6445519b60afe42abd2"} Nov 01 23:05:54.561812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.561782 5778 scope.go:122] "RemoveContainer" containerID="41e807661f50ea4049dbf76638b41c983313b168c5ff7e76903c104e409b156c" Nov 01 23:05:54.582478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.582450 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:54.588922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.588897 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-79759d4bc9-bczmc"] Nov 01 23:05:54.597596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.597573 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:54.602419 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.602399 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-5fc8dfb557-zj8zv"] Nov 01 23:05:54.633931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.623219 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:54.633931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:54.623314 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:55.448351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:55.448300 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:05:55.631885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:55.621248 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:55.631885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:55.621313 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:56.397837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:56.396970 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17741d3a-1ef0-4a47-81f3-1902f5f41879" path="/var/lib/kubelet/pods/17741d3a-1ef0-4a47-81f3-1902f5f41879/volumes" Nov 01 23:05:56.397837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:56.397447 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290d1cdf-8743-4e3f-8498-10adf8aa480e" path="/var/lib/kubelet/pods/290d1cdf-8743-4e3f-8498-10adf8aa480e/volumes" Nov 01 23:05:56.642878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:56.641125 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:56.642878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:56.641190 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:57.628878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:57.622291 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:05:57.628878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:57.622353 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 23:05:58.625826 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:58.625795 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:05:58.626354 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:58.626333 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:05:58.627264 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:58.627242 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:05:59.901740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.901706 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:05:59.902462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902442 5778 cpu_manager.go:414] "containerMap: removing container" podUID="17741d3a-1ef0-4a47-81f3-1902f5f41879" containerName="payload-pre-processing" Nov 01 23:05:59.902542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902462 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17741d3a-1ef0-4a47-81f3-1902f5f41879" containerName="payload-pre-processing" Nov 01 23:05:59.902542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902483 5778 cpu_manager.go:414] "containerMap: removing container" podUID="290d1cdf-8743-4e3f-8498-10adf8aa480e" containerName="payload-processing" Nov 01 23:05:59.902542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902488 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290d1cdf-8743-4e3f-8498-10adf8aa480e" containerName="payload-processing" Nov 01 23:05:59.902789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902618 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="17741d3a-1ef0-4a47-81f3-1902f5f41879" containerName="payload-pre-processing" Nov 01 23:05:59.902789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.902635 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="290d1cdf-8743-4e3f-8498-10adf8aa480e" containerName="payload-processing" Nov 01 23:05:59.908574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.908551 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:05:59.921940 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.915495 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-00185d-serving-cert\"" Nov 01 23:05:59.936496 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:05:59.936468 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:06:00.009395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.009357 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.009616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.009539 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5wd\" (UniqueName: \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.009616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.009582 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.009832 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.009808 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.114421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.113468 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.114421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.114030 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.114421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.114183 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5wd\" (UniqueName: \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.114421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.114230 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.116141 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.116110 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.119027 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.119007 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.124701 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.124671 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.126602 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.126576 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5wd\" (UniqueName: \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\") pod \"maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.240118 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.240085 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:00.579730 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.579618 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:06:00.629244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:00.629211 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" event={"ID":"e0199d0e-f878-46d9-a8c2-a5500284503b","Type":"ContainerStarted","Data":"1a371e1c6078ccd0c2f8398d5fc2c2dfde48f67d704331f3c5ca50d87aed7241"} Nov 01 23:06:01.641612 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:01.641571 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" event={"ID":"e0199d0e-f878-46d9-a8c2-a5500284503b","Type":"ContainerStarted","Data":"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04"} Nov 01 23:06:01.642269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:01.641905 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:01.663412 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:01.663359 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" podStartSLOduration=2.2294848800000002 podStartE2EDuration="2.663340214s" totalImagesPullingTime="433.855334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:01.660774634 +0000 UTC m=+2531.436761487" watchObservedRunningTime="2026-11-01 23:06:01.663340214 +0000 UTC m=+2531.439327046" Nov 01 23:06:06.213110 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:06.213070 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 23:06:07.657584 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:07.657554 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:06:12.441997 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:12.441953 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:06:12.456213 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:12.456179 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:06:13.889810 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.889725 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:06:13.895740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.895713 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.922064 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.922031 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:06:13.934630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.934602 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-337dee-openshift-default-dockercfg-tqjnv\"" Nov 01 23:06:13.958823 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.958789 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.958823 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.958835 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.958907 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.958951 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.958992 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8rp\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.959058 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.959079 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.959105 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:13.959136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:13.959123 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061142 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061206 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061270 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061330 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061386 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rp\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061486 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061517 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061562 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061590 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061666 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.061871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.061774 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.062084 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.062061 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.062151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.062073 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.062483 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.062429 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.064565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.064540 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.064897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.064875 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.073705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.073639 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.075349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.075321 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8rp\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.211026 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.210990 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:14.512111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.512079 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:06:14.512560 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:06:14.512535 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6688b26c_1109_45d1_b84b_71d69646e29d.slice/crio-5107e610ba1c68d850aa20d29d84dc2e50b2ceab4c4876ebdb31845ce18da0e3 WatchSource:0}: Error finding container 5107e610ba1c68d850aa20d29d84dc2e50b2ceab4c4876ebdb31845ce18da0e3: Status 404 returned error can't find the container with id 5107e610ba1c68d850aa20d29d84dc2e50b2ceab4c4876ebdb31845ce18da0e3 Nov 01 23:06:14.515611 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.515576 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:14.515724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.515646 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:14.515724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.515677 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:14.782146 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.782038 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 23:06:14.845551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.843823 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" event={"ID":"6688b26c-1109-45d1-b84b-71d69646e29d","Type":"ContainerStarted","Data":"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d"} Nov 01 23:06:14.845551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.843913 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" event={"ID":"6688b26c-1109-45d1-b84b-71d69646e29d","Type":"ContainerStarted","Data":"5107e610ba1c68d850aa20d29d84dc2e50b2ceab4c4876ebdb31845ce18da0e3"} Nov 01 23:06:14.937694 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:14.937630 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podStartSLOduration=1.93760949 podStartE2EDuration="1.93760949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:14.87368645 +0000 UTC m=+2544.649673275" watchObservedRunningTime="2026-11-01 23:06:14.93760949 +0000 UTC m=+2544.713596319" Nov 01 23:06:15.211904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:15.211869 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:15.213877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:15.213835 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:15.214108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:15.214079 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:16.211577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:16.211532 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:16.212047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:16.211593 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:17.221889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.215188 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:17.221889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.215267 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:17.443878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.442922 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" podUID="74447c63-f037-42d7-92c7-62ac8a753056" containerName="payload-processing" containerID="cri-o://30ff2db5df66807e3243cf860d60a510d55e724a741df1e4c820ddce97fbab31" gracePeriod=25 Nov 01 23:06:17.457099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.457048 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" podUID="293c4777-ab30-42ae-afa2-c277b2a281df" containerName="payload-pre-processing" containerID="cri-o://c176dc4748c00a30381ec308cb5d0a3341a636175b282dcb9b9ac1fda96447e2" gracePeriod=25 Nov 01 23:06:17.901246 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.901207 5778 generic.go:356] "Generic (PLEG): container finished" podID="74447c63-f037-42d7-92c7-62ac8a753056" containerID="30ff2db5df66807e3243cf860d60a510d55e724a741df1e4c820ddce97fbab31" exitCode=0 Nov 01 23:06:17.901411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.901332 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" event={"ID":"74447c63-f037-42d7-92c7-62ac8a753056","Type":"ContainerDied","Data":"30ff2db5df66807e3243cf860d60a510d55e724a741df1e4c820ddce97fbab31"} Nov 01 23:06:17.901411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.901395 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" event={"ID":"74447c63-f037-42d7-92c7-62ac8a753056","Type":"ContainerDied","Data":"09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be"} Nov 01 23:06:17.901411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.901411 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b9d08d173e9c6907d8d3b83def24c5fd0e79a197837d205b808e3672fad8be" Nov 01 23:06:17.910495 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.910456 5778 generic.go:356] "Generic (PLEG): container finished" podID="293c4777-ab30-42ae-afa2-c277b2a281df" containerID="c176dc4748c00a30381ec308cb5d0a3341a636175b282dcb9b9ac1fda96447e2" exitCode=0 Nov 01 23:06:17.910612 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.910526 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" event={"ID":"293c4777-ab30-42ae-afa2-c277b2a281df","Type":"ContainerDied","Data":"c176dc4748c00a30381ec308cb5d0a3341a636175b282dcb9b9ac1fda96447e2"} Nov 01 23:06:17.933878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:17.922224 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:06:18.068872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.068607 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:06:18.102910 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.102871 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\") pod \"293c4777-ab30-42ae-afa2-c277b2a281df\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " Nov 01 23:06:18.103112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.102974 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\") pod \"74447c63-f037-42d7-92c7-62ac8a753056\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " Nov 01 23:06:18.103112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.103004 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\" (UniqueName: \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\") pod \"293c4777-ab30-42ae-afa2-c277b2a281df\" (UID: \"293c4777-ab30-42ae-afa2-c277b2a281df\") " Nov 01 23:06:18.103112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.103068 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\" (UniqueName: \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\") pod \"74447c63-f037-42d7-92c7-62ac8a753056\" (UID: \"74447c63-f037-42d7-92c7-62ac8a753056\") " Nov 01 23:06:18.104577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.104520 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume" pod "293c4777-ab30-42ae-afa2-c277b2a281df" (UID: "293c4777-ab30-42ae-afa2-c277b2a281df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:18.104916 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.104886 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume" pod "74447c63-f037-42d7-92c7-62ac8a753056" (UID: "74447c63-f037-42d7-92c7-62ac8a753056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:18.106968 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.106936 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz" pod "74447c63-f037-42d7-92c7-62ac8a753056" (UID: "74447c63-f037-42d7-92c7-62ac8a753056"). InnerVolumeSpecName "kube-api-access-pstfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:18.107393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.107369 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv" pod "293c4777-ab30-42ae-afa2-c277b2a281df" (UID: "293c4777-ab30-42ae-afa2-c277b2a281df"). InnerVolumeSpecName "kube-api-access-czpvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:18.204740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.204704 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstfz\" (UniqueName: \"kubernetes.io/projected/74447c63-f037-42d7-92c7-62ac8a753056-kube-api-access-pstfz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:18.204740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.204738 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c4777-ab30-42ae-afa2-c277b2a281df-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:18.204740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.204749 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74447c63-f037-42d7-92c7-62ac8a753056-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:18.204740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.204757 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czpvv\" (UniqueName: \"kubernetes.io/projected/293c4777-ab30-42ae-afa2-c277b2a281df-kube-api-access-czpvv\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:18.212366 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.212329 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:18.212521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.212400 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:18.776577 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:18.776501 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:06:18.949478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.949444 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" Nov 01 23:06:18.949782 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.949473 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd" event={"ID":"293c4777-ab30-42ae-afa2-c277b2a281df","Type":"ContainerDied","Data":"3d745b2c9c124bbd6cb125bc485763d56dde7d727015bbf1070a29bc11ee2705"} Nov 01 23:06:18.949908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.949829 5778 scope.go:122] "RemoveContainer" containerID="c176dc4748c00a30381ec308cb5d0a3341a636175b282dcb9b9ac1fda96447e2" Nov 01 23:06:18.950240 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.949507 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm" Nov 01 23:06:18.976211 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.976180 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:06:18.980334 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.980310 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-5798c865f5-z2jcd"] Nov 01 23:06:18.992313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.992282 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:06:18.995685 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:18.995663 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-5d9596f5bc-4dmxm"] Nov 01 23:06:19.212168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:19.212131 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:19.212339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:19.212188 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:20.211457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:20.211420 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:20.211874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:20.211470 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:20.376306 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:20.376269 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293c4777-ab30-42ae-afa2-c277b2a281df" path="/var/lib/kubelet/pods/293c4777-ab30-42ae-afa2-c277b2a281df/volumes" Nov 01 23:06:20.376724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:20.376710 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74447c63-f037-42d7-92c7-62ac8a753056" path="/var/lib/kubelet/pods/74447c63-f037-42d7-92c7-62ac8a753056/volumes" Nov 01 23:06:21.212221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.212184 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:06:21.212695 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.212251 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 23:06:21.260723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.260677 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerName="main" containerID="cri-o://6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f" gracePeriod=45 Nov 01 23:06:21.442489 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.442434 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": dial tcp 10.132.0.163:8000: connect: connection refused" Nov 01 23:06:21.609064 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.609035 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 23:06:21.695925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.695894 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.695948 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.695979 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696146 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\" (UniqueName: \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696191 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696306 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696212 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\") pod \"e4cce74b-0093-4df5-b5e7-200b8da48ced\" (UID: \"e4cce74b-0093-4df5-b5e7-200b8da48ced\") " Nov 01 23:06:21.696368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696324 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:21.696500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696442 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:21.696585 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696563 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:21.696831 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696812 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.696949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696836 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.696949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.696889 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.698540 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.698516 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:21.698638 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.698562 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "kube-api-access-d7m4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:21.698638 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.698573 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs" pod "e4cce74b-0093-4df5-b5e7-200b8da48ced" (UID: "e4cce74b-0093-4df5-b5e7-200b8da48ced"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:21.797947 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.797826 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7m4p\" (UniqueName: \"kubernetes.io/projected/e4cce74b-0093-4df5-b5e7-200b8da48ced-kube-api-access-d7m4p\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.797947 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.797904 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cce74b-0093-4df5-b5e7-200b8da48ced-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.797947 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.797915 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cce74b-0093-4df5-b5e7-200b8da48ced-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:21.977043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.977006 5778 generic.go:356] "Generic (PLEG): container finished" podID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerID="6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f" exitCode=0 Nov 01 23:06:21.977212 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.977058 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" Nov 01 23:06:21.977212 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.977097 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" event={"ID":"e4cce74b-0093-4df5-b5e7-200b8da48ced","Type":"ContainerDied","Data":"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f"} Nov 01 23:06:21.977212 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.977148 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr" event={"ID":"e4cce74b-0093-4df5-b5e7-200b8da48ced","Type":"ContainerDied","Data":"1c901d223a7f5e4e0616635faf43e3deb1a99a697980de74d132715af57e7daa"} Nov 01 23:06:21.977212 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.977172 5778 scope.go:122] "RemoveContainer" containerID="6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f" Nov 01 23:06:21.989608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.989590 5778 scope.go:122] "RemoveContainer" containerID="6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f" Nov 01 23:06:21.989945 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:21.989925 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f\": container with ID starting with 6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f not found: ID does not exist" containerID="6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f" Nov 01 23:06:21.990001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:21.989954 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f"} err="failed to get container status \"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f\": rpc error: code = NotFound desc = could not find container \"6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f\": container with ID starting with 6ba503c1e2328440223aff1b3b20bfc308aa2bb16ecbd31fcc546a5e47ab100f not found: ID does not exist" Nov 01 23:06:22.005505 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.005467 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 23:06:22.015067 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.015038 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/auth-test-model-7366bf-kserve-88cb49ddf-lshtr"] Nov 01 23:06:22.218551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.218517 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:22.219321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.219300 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:22.220172 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.220149 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:06:22.383193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:22.383156 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" path="/var/lib/kubelet/pods/e4cce74b-0093-4df5-b5e7-200b8da48ced/volumes" Nov 01 23:06:25.033185 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:25.033149 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.307690 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309304 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerName="main" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309328 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerName="main" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309402 5778 cpu_manager.go:414] "containerMap: removing container" podUID="293c4777-ab30-42ae-afa2-c277b2a281df" containerName="payload-pre-processing" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309413 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293c4777-ab30-42ae-afa2-c277b2a281df" containerName="payload-pre-processing" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309433 5778 cpu_manager.go:414] "containerMap: removing container" podUID="74447c63-f037-42d7-92c7-62ac8a753056" containerName="payload-processing" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309458 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74447c63-f037-42d7-92c7-62ac8a753056" containerName="payload-processing" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309767 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cce74b-0093-4df5-b5e7-200b8da48ced" containerName="main" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309793 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="293c4777-ab30-42ae-afa2-c277b2a281df" containerName="payload-pre-processing" Nov 01 23:06:27.310789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.309822 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="74447c63-f037-42d7-92c7-62ac8a753056" containerName="payload-processing" Nov 01 23:06:27.321822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.320441 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.329301 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.329273 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"openshift-service-ca.crt\"" Nov 01 23:06:27.334655 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.334628 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:06:27.370870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.370238 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"default-dockercfg-l55dr\"" Nov 01 23:06:27.370870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.370236 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"kube-root-ca.crt\"" Nov 01 23:06:27.370870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.370302 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-7366bf\"/\"sub-test-model-7366bf-kserve-self-signed-certs\"" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473250 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473303 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473394 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473443 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqmd\" (UniqueName: \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473579 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.488875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.473650 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.575910 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.576108 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.576145 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.576204 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.576244 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqmd\" (UniqueName: \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.576934 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.576400 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.577436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.577042 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.580870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.577697 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.580870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.577967 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.604875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.603740 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.604875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.604418 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.605130 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.605068 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqmd\" (UniqueName: \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\") pod \"sub-test-model-7366bf-kserve-645d9b97bf-44b47\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.677254 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.676955 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:27.986839 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:27.986808 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:06:27.988027 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:06:27.987999 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a9d05f7_c83d_4432_a138_5070de7f3424.slice/crio-98f18ac02ab73a28be09594e22674fa766d76b3636b7742b3275910d7c1ca377 WatchSource:0}: Error finding container 98f18ac02ab73a28be09594e22674fa766d76b3636b7742b3275910d7c1ca377: Status 404 returned error can't find the container with id 98f18ac02ab73a28be09594e22674fa766d76b3636b7742b3275910d7c1ca377 Nov 01 23:06:28.039301 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:28.039266 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" event={"ID":"6a9d05f7-c83d-4432-a138-5070de7f3424","Type":"ContainerStarted","Data":"98f18ac02ab73a28be09594e22674fa766d76b3636b7742b3275910d7c1ca377"} Nov 01 23:06:29.050904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:29.050836 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" event={"ID":"6a9d05f7-c83d-4432-a138-5070de7f3424","Type":"ContainerStarted","Data":"22a420fd2b66821c76a15c45bb3d221e76ce7968b1eabd62a7df490e17213946"} Nov 01 23:06:29.070108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:29.070054 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" podStartSLOduration=2.070039999 podStartE2EDuration="2.070039999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:29.068840892 +0000 UTC m=+2558.844827743" watchObservedRunningTime="2026-11-01 23:06:29.070039999 +0000 UTC m=+2558.846026828" Nov 01 23:06:29.678275 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:29.678244 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:29.812481 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:29.812436 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" podUID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" containerName="main" containerID="cri-o://192bf58fd814f56f4bddbfc74ccc69e737355f5443369f7a5f16079612bb80e3" gracePeriod=45 Nov 01 23:06:30.034127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.034073 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" podUID="7720b68e-9062-45ee-bf34-a240f4b9c13f" containerName="payload-processing" containerID="cri-o://12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762" gracePeriod=25 Nov 01 23:06:30.070500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.070466 5778 generic.go:356] "Generic (PLEG): container finished" podID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" containerID="192bf58fd814f56f4bddbfc74ccc69e737355f5443369f7a5f16079612bb80e3" exitCode=0 Nov 01 23:06:30.070978 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.070819 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" event={"ID":"d0462f7f-2b3b-46af-9638-b6010c9e7e6d","Type":"ContainerDied","Data":"192bf58fd814f56f4bddbfc74ccc69e737355f5443369f7a5f16079612bb80e3"} Nov 01 23:06:30.176900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.175393 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245322 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\" (UniqueName: \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245495 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245541 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245570 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245656 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.245735 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\") pod \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\" (UID: \"d0462f7f-2b3b-46af-9638-b6010c9e7e6d\") " Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.246371 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.247014 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:30.249154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.247247 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:30.251199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.251173 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "kube-api-access-hbtvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:30.252428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.252403 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:30.264055 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.264023 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm" pod "d0462f7f-2b3b-46af-9638-b6010c9e7e6d" (UID: "d0462f7f-2b3b-46af-9638-b6010c9e7e6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:30.348028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.347990 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.348028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.348028 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.348718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.348044 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.348718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.348059 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.348718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.348079 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.348718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.348092 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtvg\" (UniqueName: \"kubernetes.io/projected/d0462f7f-2b3b-46af-9638-b6010c9e7e6d-kube-api-access-hbtvg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.391950 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.391916 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:06:30.555142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.555040 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\" (UniqueName: \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\") pod \"7720b68e-9062-45ee-bf34-a240f4b9c13f\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " Nov 01 23:06:30.555142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.555122 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\") pod \"7720b68e-9062-45ee-bf34-a240f4b9c13f\" (UID: \"7720b68e-9062-45ee-bf34-a240f4b9c13f\") " Nov 01 23:06:30.555974 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.555805 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume" pod "7720b68e-9062-45ee-bf34-a240f4b9c13f" (UID: "7720b68e-9062-45ee-bf34-a240f4b9c13f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:30.556197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.556083 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7720b68e-9062-45ee-bf34-a240f4b9c13f-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.558735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.558702 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5" pod "7720b68e-9062-45ee-bf34-a240f4b9c13f" (UID: "7720b68e-9062-45ee-bf34-a240f4b9c13f"). InnerVolumeSpecName "kube-api-access-g9xf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:30.657587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:30.657549 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xf5\" (UniqueName: \"kubernetes.io/projected/7720b68e-9062-45ee-bf34-a240f4b9c13f-kube-api-access-g9xf5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.079782 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.079748 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" Nov 01 23:06:31.080403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.079779 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9" event={"ID":"d0462f7f-2b3b-46af-9638-b6010c9e7e6d","Type":"ContainerDied","Data":"8ec535fa07bf73c5687e97e5b17ca43d5db58afd3787ef5e2f6bb52c6d532b40"} Nov 01 23:06:31.080403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.079838 5778 scope.go:122] "RemoveContainer" containerID="192bf58fd814f56f4bddbfc74ccc69e737355f5443369f7a5f16079612bb80e3" Nov 01 23:06:31.081520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.081494 5778 generic.go:356] "Generic (PLEG): container finished" podID="7720b68e-9062-45ee-bf34-a240f4b9c13f" containerID="12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762" exitCode=0 Nov 01 23:06:31.081663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.081543 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" Nov 01 23:06:31.082813 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.082788 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" event={"ID":"7720b68e-9062-45ee-bf34-a240f4b9c13f","Type":"ContainerDied","Data":"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762"} Nov 01 23:06:31.082954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.082836 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl" event={"ID":"7720b68e-9062-45ee-bf34-a240f4b9c13f","Type":"ContainerDied","Data":"350f42439f8b674581ae60ba9488a06f3412e58f0ba9bef510018f34c3cc1de8"} Nov 01 23:06:31.091098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.091067 5778 scope.go:122] "RemoveContainer" containerID="12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762" Nov 01 23:06:31.099542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.099521 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 23:06:31.103590 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.103567 5778 scope.go:122] "RemoveContainer" containerID="12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762" Nov 01 23:06:31.103980 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:31.103958 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762\": container with ID starting with 12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762 not found: ID does not exist" containerID="12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762" Nov 01 23:06:31.104080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.103992 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762"} err="failed to get container status \"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762\": rpc error: code = NotFound desc = could not find container \"12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762\": container with ID starting with 12f120c083af096c1b88520432af05c696643c86128f1c1c6e9acac37339b762 not found: ID does not exist" Nov 01 23:06:31.105146 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.105125 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/auth-test-model-075771-kserve-95bdd65df-rbgk9"] Nov 01 23:06:31.119097 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.119072 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:06:31.126493 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:31.126471 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-56c7f9969f-vqnwl"] Nov 01 23:06:32.421879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:32.420014 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7720b68e-9062-45ee-bf34-a240f4b9c13f" path="/var/lib/kubelet/pods/7720b68e-9062-45ee-bf34-a240f4b9c13f/volumes" Nov 01 23:06:32.421879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:32.420722 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" path="/var/lib/kubelet/pods/d0462f7f-2b3b-46af-9638-b6010c9e7e6d/volumes" Nov 01 23:06:37.681988 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:37.681957 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:37.706133 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:37.706098 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:38.141740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.141642 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-ff48b3"] Nov 01 23:06:38.144238 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.144208 5778 cpu_manager.go:414] "containerMap: removing container" podUID="7720b68e-9062-45ee-bf34-a240f4b9c13f" containerName="payload-processing" Nov 01 23:06:38.144432 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.144418 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7720b68e-9062-45ee-bf34-a240f4b9c13f" containerName="payload-processing" Nov 01 23:06:38.144570 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.144560 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" containerName="main" Nov 01 23:06:38.144654 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.144644 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" containerName="main" Nov 01 23:06:38.152411 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.152382 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0462f7f-2b3b-46af-9638-b6010c9e7e6d" containerName="main" Nov 01 23:06:38.152698 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.152678 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="7720b68e-9062-45ee-bf34-a240f4b9c13f" containerName="payload-processing" Nov 01 23:06:38.178399 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.178365 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ff48b3"] Nov 01 23:06:38.178763 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.178701 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:38.196980 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.191722 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bf947\"" Nov 01 23:06:38.206880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.204988 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2bh\" (UniqueName: \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\") pod \"test-curl-6968-ff48b3\" (UID: \"a4abbb49-41fd-4ee4-a078-07f8eb65c671\") " pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:38.312873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.306718 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2bh\" (UniqueName: \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\") pod \"test-curl-6968-ff48b3\" (UID: \"a4abbb49-41fd-4ee4-a078-07f8eb65c671\") " pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:38.360875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.359730 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2bh\" (UniqueName: \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\") pod \"test-curl-6968-ff48b3\" (UID: \"a4abbb49-41fd-4ee4-a078-07f8eb65c671\") " pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:38.506019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.505976 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:38.513003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:38.512967 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:39.291769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:39.291002 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-ff48b3"] Nov 01 23:06:39.311771 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:39.311737 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ff48b3" event={"ID":"a4abbb49-41fd-4ee4-a078-07f8eb65c671","Type":"ContainerStarted","Data":"c34d45a77ba9078d499d6f37f04fe1c25fb6fb2b15b6be8c7b3028f66869db15"} Nov 01 23:06:40.344496 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:40.344458 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-ff48b3" event={"ID":"a4abbb49-41fd-4ee4-a078-07f8eb65c671","Type":"ContainerStarted","Data":"f77d59fbdda9462959aec01f91e1726711fb26c6b7f653411e68191300be6df1"} Nov 01 23:06:40.364961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:40.364918 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-ff48b3" podStartSLOduration=2.364899991 podStartE2EDuration="2.364899991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:40.364684837 +0000 UTC m=+2570.140671672" watchObservedRunningTime="2026-11-01 23:06:40.364899991 +0000 UTC m=+2570.140886821" Nov 01 23:06:41.017693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.017610 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-ff48b3"] Nov 01 23:06:41.026655 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.026608 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-ff48b3"] Nov 01 23:06:41.139492 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.139449 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-fa5633"] Nov 01 23:06:41.140270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.140247 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" containerName="test-curl-6968-ff48b3" Nov 01 23:06:41.140270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.140269 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" containerName="test-curl-6968-ff48b3" Nov 01 23:06:41.140513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.140497 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" containerName="test-curl-6968-ff48b3" Nov 01 23:06:41.144504 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.144485 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:41.151356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.151324 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-fa5633"] Nov 01 23:06:41.272230 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.272137 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7k2\" (UniqueName: \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\") pod \"test-curl-6968-fa5633\" (UID: \"41e70285-13a5-4e77-9301-13b6809d2824\") " pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:41.373328 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.373294 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7k2\" (UniqueName: \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\") pod \"test-curl-6968-fa5633\" (UID: \"41e70285-13a5-4e77-9301-13b6809d2824\") " pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:41.381716 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.381686 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7k2\" (UniqueName: \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\") pod \"test-curl-6968-fa5633\" (UID: \"41e70285-13a5-4e77-9301-13b6809d2824\") " pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:41.465608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.465575 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:41.777446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:41.777389 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-fa5633"] Nov 01 23:06:41.778113 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:06:41.777744 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41e70285_13a5_4e77_9301_13b6809d2824.slice/crio-c7bba15ff141537ca04d602283012ddb3679759de82ae6e01394e10cc65cc772 WatchSource:0}: Error finding container c7bba15ff141537ca04d602283012ddb3679759de82ae6e01394e10cc65cc772: Status 404 returned error can't find the container with id c7bba15ff141537ca04d602283012ddb3679759de82ae6e01394e10cc65cc772 Nov 01 23:06:42.373066 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:42.373020 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-ff48b3" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" containerName="test-curl-6968-ff48b3" containerID="cri-o://f77d59fbdda9462959aec01f91e1726711fb26c6b7f653411e68191300be6df1" gracePeriod=2 Nov 01 23:06:42.377894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:42.377825 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-fa5633" event={"ID":"41e70285-13a5-4e77-9301-13b6809d2824","Type":"ContainerStarted","Data":"13b6d83bc284e03c7f59a988fbe0e6d47c3f8f2ec6bdff2b2547794f376673d6"} Nov 01 23:06:42.377894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:42.377891 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-fa5633" event={"ID":"41e70285-13a5-4e77-9301-13b6809d2824","Type":"ContainerStarted","Data":"c7bba15ff141537ca04d602283012ddb3679759de82ae6e01394e10cc65cc772"} Nov 01 23:06:42.388999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:42.388948 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-fa5633" podStartSLOduration=1.388930419 podStartE2EDuration="1.388930419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:42.388208872 +0000 UTC m=+2572.164195703" watchObservedRunningTime="2026-11-01 23:06:42.388930419 +0000 UTC m=+2572.164917275" Nov 01 23:06:42.391566 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:42.390902 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ff48b3\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:43.026315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.026216 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-fa5633"] Nov 01 23:06:43.037841 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.033832 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-fa5633"] Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.114367 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.115455 5778 cpu_manager.go:414] "containerMap: removing container" podUID="41e70285-13a5-4e77-9301-13b6809d2824" containerName="test-curl-6968-fa5633" Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.115478 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e70285-13a5-4e77-9301-13b6809d2824" containerName="test-curl-6968-fa5633" Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.115742 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e70285-13a5-4e77-9301-13b6809d2824" containerName="test-curl-6968-fa5633" Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.121181 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.126929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.125427 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:06:43.127412 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.127029 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"openshift-service-ca.crt\"" Nov 01 23:06:43.141919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.128843 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"sub-test-model-075771-kserve-self-signed-certs\"" Nov 01 23:06:43.141919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.129017 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"default-dockercfg-866mq\"" Nov 01 23:06:43.141919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.129078 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-075771\"/\"kube-root-ca.crt\"" Nov 01 23:06:43.212455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212419 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.212661 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212616 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.212742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212665 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7xc\" (UniqueName: \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.212742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212694 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.212742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212730 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.212942 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.212900 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314516 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314595 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314821 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314890 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7xc\" (UniqueName: \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314928 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.314964 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.315521 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.316295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.315786 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.326211 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.325545 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.365900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.354709 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.430895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.421948 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.468892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.459670 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7xc\" (UniqueName: \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\") pod \"sub-test-model-075771-kserve-659c4cdfdc-j4llw\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:43.746105 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:43.746063 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:44.089479 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.088714 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:06:44.471182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.471143 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" event={"ID":"bf7ae301-5191-4511-912d-fb6bdb280530","Type":"ContainerStarted","Data":"b5e73513794edc4744c5331e0dd495b570de11649364c1bdfeb09a7df7ff6fac"} Nov 01 23:06:44.471182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.471189 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" event={"ID":"bf7ae301-5191-4511-912d-fb6bdb280530","Type":"ContainerStarted","Data":"c4317dddc9c556f58217df806bb47589b3204aaad92644d6c496e7b6258487ac"} Nov 01 23:06:44.474219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.474187 5778 generic.go:356] "Generic (PLEG): container finished" podID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" containerID="f77d59fbdda9462959aec01f91e1726711fb26c6b7f653411e68191300be6df1" exitCode=137 Nov 01 23:06:44.474455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.474427 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-fa5633" podUID="41e70285-13a5-4e77-9301-13b6809d2824" containerName="test-curl-6968-fa5633" containerID="cri-o://13b6d83bc284e03c7f59a988fbe0e6d47c3f8f2ec6bdff2b2547794f376673d6" gracePeriod=2 Nov 01 23:06:44.496722 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.496651 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" podStartSLOduration=1.496627259 podStartE2EDuration="1.496627259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:44.486675378 +0000 UTC m=+2574.262662207" watchObservedRunningTime="2026-11-01 23:06:44.496627259 +0000 UTC m=+2574.272614090" Nov 01 23:06:44.652840 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.652763 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:44.755198 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.753694 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\" (UniqueName: \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\") pod \"a4abbb49-41fd-4ee4-a078-07f8eb65c671\" (UID: \"a4abbb49-41fd-4ee4-a078-07f8eb65c671\") " Nov 01 23:06:44.755198 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.753933 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:44.773776 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.773237 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh" pod "a4abbb49-41fd-4ee4-a078-07f8eb65c671" (UID: "a4abbb49-41fd-4ee4-a078-07f8eb65c671"). InnerVolumeSpecName "kube-api-access-wv2bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:44.856335 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:44.856303 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2bh\" (UniqueName: \"kubernetes.io/projected/a4abbb49-41fd-4ee4-a078-07f8eb65c671-kube-api-access-wv2bh\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:45.509678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:45.509646 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:45.556876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:45.525023 5778 scope.go:122] "RemoveContainer" containerID="f77d59fbdda9462959aec01f91e1726711fb26c6b7f653411e68191300be6df1" Nov 01 23:06:45.580904 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:45.578681 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-ff48b3\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" pod="opendatahub/test-curl-6968-ff48b3" Nov 01 23:06:46.396993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.391764 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4abbb49-41fd-4ee4-a078-07f8eb65c671" path="/var/lib/kubelet/pods/a4abbb49-41fd-4ee4-a078-07f8eb65c671/volumes" Nov 01 23:06:46.522692 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.522655 5778 generic.go:356] "Generic (PLEG): container finished" podID="41e70285-13a5-4e77-9301-13b6809d2824" containerID="13b6d83bc284e03c7f59a988fbe0e6d47c3f8f2ec6bdff2b2547794f376673d6" exitCode=137 Nov 01 23:06:46.535909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.535832 5778 generic.go:356] "Generic (PLEG): container finished" podID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerID="bd275253900f9a5311ec3e4ea3eaea18f136f912e717c90cd114627bae5117ed" exitCode=0 Nov 01 23:06:46.536148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.536121 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerDied","Data":"bd275253900f9a5311ec3e4ea3eaea18f136f912e717c90cd114627bae5117ed"} Nov 01 23:06:46.536909 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.536645 5778 scope.go:122] "RemoveContainer" containerID="bd275253900f9a5311ec3e4ea3eaea18f136f912e717c90cd114627bae5117ed" Nov 01 23:06:46.542132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.541786 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.542132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.541876 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.542132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.541920 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.550919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.550725 5778 generic.go:356] "Generic (PLEG): container finished" podID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerID="4ef042699db460dcb3bef641267fcdde94571145a2ea38350172454bd2dbc715" exitCode=0 Nov 01 23:06:46.551174 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.551104 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerDied","Data":"4ef042699db460dcb3bef641267fcdde94571145a2ea38350172454bd2dbc715"} Nov 01 23:06:46.552198 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.552117 5778 scope.go:122] "RemoveContainer" containerID="4ef042699db460dcb3bef641267fcdde94571145a2ea38350172454bd2dbc715" Nov 01 23:06:46.572900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.571509 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.572900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.571583 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.572900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.571623 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:46.709231 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.709086 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:46.818016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.817977 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\" (UniqueName: \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\") pod \"41e70285-13a5-4e77-9301-13b6809d2824\" (UID: \"41e70285-13a5-4e77-9301-13b6809d2824\") " Nov 01 23:06:46.822462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.822430 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2" pod "41e70285-13a5-4e77-9301-13b6809d2824" (UID: "41e70285-13a5-4e77-9301-13b6809d2824"). InnerVolumeSpecName "kube-api-access-vg7k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:46.921559 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:46.921472 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7k2\" (UniqueName: \"kubernetes.io/projected/41e70285-13a5-4e77-9301-13b6809d2824-kube-api-access-vg7k2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:47.478184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.478138 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:06:47.659871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.659641 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerStarted","Data":"b89b30b98998c294b84ea516bd2e2f1502e4d7db0de8d20a2b0acf6d19069ba5"} Nov 01 23:06:47.702178 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.700829 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerStarted","Data":"2dc95e1f5895ed2f889fb9d91d57250301a7b1597f659ca5966729c7bbfedc37"} Nov 01 23:06:47.704622 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.704182 5778 scope.go:122] "RemoveContainer" containerID="13b6d83bc284e03c7f59a988fbe0e6d47c3f8f2ec6bdff2b2547794f376673d6" Nov 01 23:06:47.704622 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.704297 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:47.743116 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:47.739958 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-fa5633\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="41e70285-13a5-4e77-9301-13b6809d2824" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:47.804813 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.804784 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:06:47.807101 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.807077 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:47.807277 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:47.807136 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:47.857448 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:47.857407 5778 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41e70285_13a5_4e77_9301_13b6809d2824.slice\": RecentStats: unable to find data in memory cache]" Nov 01 23:06:48.198154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.198116 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:06:48.204102 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.204074 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.206676 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:48.206638 5778 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-fa5633\" is forbidden: User \"system:node:ip-10-0-141-223.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-223.us-east-2.compute.internal' and this object" podUID="41e70285-13a5-4e77-9301-13b6809d2824" pod="opendatahub/test-curl-6968-fa5633" Nov 01 23:06:48.209069 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.209046 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-56b833-openshift-default-dockercfg-grknh\"" Nov 01 23:06:48.218877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.214975 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:06:48.390704 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.390661 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.391223 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.391194 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.392216 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.392194 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.392938 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.392918 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.393234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.393215 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.394166 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.394146 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n92k\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.394589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.394570 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.394800 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.394784 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.394969 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.394954 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.494120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.494025 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:48.494399 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.494368 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:48.495542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.495510 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e70285-13a5-4e77-9301-13b6809d2824" path="/var/lib/kubelet/pods/41e70285-13a5-4e77-9301-13b6809d2824/volumes" Nov 01 23:06:48.495984 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.495968 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:06:48.496738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.496709 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.496757 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n92k\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.496793 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.496930 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.496957 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497068 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497142 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497212 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497246 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497278 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.497807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.497782 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.498045 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.498025 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.498156 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.498044 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.498478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.498457 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.500375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.500354 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.503191 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.503165 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.505546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.505517 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n92k\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.506001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.505984 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.586200 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.586160 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:48.806592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.806163 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:48.806592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.806220 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:48.893656 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:06:48.893585 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:06:48.990830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:48.990793 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:06:49.001998 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:06:49.001947 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ffb239_7a92_4b4a_bd57_30c90c3d2942.slice/crio-674cc16d8627588155da98f279a6fdd8a2b9eae82affbd5663407ecf7af0a898 WatchSource:0}: Error finding container 674cc16d8627588155da98f279a6fdd8a2b9eae82affbd5663407ecf7af0a898: Status 404 returned error can't find the container with id 674cc16d8627588155da98f279a6fdd8a2b9eae82affbd5663407ecf7af0a898 Nov 01 23:06:49.004599 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.004319 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:49.004599 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.004461 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:49.004599 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.004508 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:49.418964 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.418926 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:49.419244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.419211 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:49.792294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.792256 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" event={"ID":"e3ffb239-7a92-4b4a-bd57-30c90c3d2942","Type":"ContainerStarted","Data":"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08"} Nov 01 23:06:49.792612 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.792585 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" event={"ID":"e3ffb239-7a92-4b4a-bd57-30c90c3d2942","Type":"ContainerStarted","Data":"674cc16d8627588155da98f279a6fdd8a2b9eae82affbd5663407ecf7af0a898"} Nov 01 23:06:49.805118 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.805085 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:49.805264 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.805135 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:49.820712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:49.820653 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podStartSLOduration=1.8206347090000001 podStartE2EDuration="1.820634709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:49.814797361 +0000 UTC m=+2579.590784200" watchObservedRunningTime="2026-11-01 23:06:49.820634709 +0000 UTC m=+2579.596621542" Nov 01 23:06:50.607675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.605401 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:06:50.607675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.605516 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:50.607675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.605556 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:50.640239 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.640196 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:50.640440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.640274 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:50.844090 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.844047 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:50.844703 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.844666 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:50.933932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.912733 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:06:50.933932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:50.913008 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerName="maas-api" containerID="cri-o://7b466934cb2227de208a95dcf43aacb940f3596ced24895908aafc99096ac73b" gracePeriod=30 Nov 01 23:06:51.327996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.293941 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:51.472887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.472276 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:51.562886 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.556213 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:51.703434 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.666763 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:51.707753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.707713 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:51.708162 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.708127 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:51.727452 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.727405 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.177:8443/health\": dial tcp 10.132.0.177:8443: connect: connection refused" Nov 01 23:06:51.768428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.768379 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 01 23:06:51.986940 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.984965 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:51.986940 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:51.985040 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:52.021881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.021773 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": context deadline exceeded" Nov 01 23:06:52.170236 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.170193 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 23:06:52.310692 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.310545 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:52.313048 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.312997 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:52.369902 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.314842 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:06:52.486877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.450055 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:52.718925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.499958 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded" Nov 01 23:06:52.719911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.539553 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:52.719911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.718124 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:52.719911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.636827 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/health\": context deadline exceeded" Nov 01 23:06:52.721159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.721119 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:52.892392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.892345 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:53.192805 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:52.916489 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:53.195525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.098148 5778 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:53.201072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.133949 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:53.201295 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.201268 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:53.201397 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.179022 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 23:06:53.202169 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.192994 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:53.202340 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.134139 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": context deadline exceeded" Nov 01 23:06:53.209869 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.207673 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:06:53.239132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.235156 5778 generic.go:356] "Generic (PLEG): container finished" podID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerID="7b466934cb2227de208a95dcf43aacb940f3596ced24895908aafc99096ac73b" exitCode=0 Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292531 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292657 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292709 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292726 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" event={"ID":"a58fae54-0661-4c07-9c3e-3e4329b24def","Type":"ContainerDied","Data":"7b466934cb2227de208a95dcf43aacb940f3596ced24895908aafc99096ac73b"} Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292764 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292805 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292841 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292893 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292926 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:06:53.299880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.292955 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:06:53.312036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.312008 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.350776 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.349070 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dc635e-openshift-default-dockercfg-fjhdk\"" Nov 01 23:06:53.359075 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.359042 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:06:53.392437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.392404 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.431011 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.431084 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.432038 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\") pod \"a58fae54-0661-4c07-9c3e-3e4329b24def\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.432141 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\") pod \"a58fae54-0661-4c07-9c3e-3e4329b24def\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.432489 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\" (UniqueName: \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\") pod \"a58fae54-0661-4c07-9c3e-3e4329b24def\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.440812 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\") pod \"a58fae54-0661-4c07-9c3e-3e4329b24def\" (UID: \"a58fae54-0661-4c07-9c3e-3e4329b24def\") " Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.444584 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca" pod "a58fae54-0661-4c07-9c3e-3e4329b24def" (UID: "a58fae54-0661-4c07-9c3e-3e4329b24def"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.461309 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.477833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.461402 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.565047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.565009 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs" pod "a58fae54-0661-4c07-9c3e-3e4329b24def" (UID: "a58fae54-0661-4c07-9c3e-3e4329b24def"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:53.615107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.565286 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.615575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.615544 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.615763 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.615743 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.615920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.615903 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.616283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.616262 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mr5j\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.616434 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.616419 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.616551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.616537 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.616820 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.616803 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58fae54-0661-4c07-9c3e-3e4329b24def-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:53.616963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.616949 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:53.625881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.625093 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k" pod "a58fae54-0661-4c07-9c3e-3e4329b24def" (UID: "a58fae54-0661-4c07-9c3e-3e4329b24def"). InnerVolumeSpecName "kube-api-access-jpv6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:53.653174 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.652818 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls" pod "a58fae54-0661-4c07-9c3e-3e4329b24def" (UID: "a58fae54-0661-4c07-9c3e-3e4329b24def"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:53.656043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.653517 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:53.656043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.653589 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:53.656043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.654661 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:53.656043 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.655184 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 23:06:53.737294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.736913 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mr5j\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.737572 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.737552 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.743974 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.743838 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.744193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.743934 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.760462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.760432 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.761075 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.761052 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.778682 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.778643 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.779092 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.779062 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.779278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.779260 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.779407 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.779389 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.779647 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.779631 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58fae54-0661-4c07-9c3e-3e4329b24def-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:53.779825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.779810 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpv6k\" (UniqueName: \"kubernetes.io/projected/a58fae54-0661-4c07-9c3e-3e4329b24def-kube-api-access-jpv6k\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:53.780731 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.780704 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.781779 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.758954 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:06:53.782018 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.760981 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.782167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.781465 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.782283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.781703 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:53.806661 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.806627 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:53.807009 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.806967 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:53.892221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:53.892177 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:54.002948 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.002227 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 23:06:54.058086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.052172 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:54.231690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.196438 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 23:06:54.231690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.231607 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mr5j\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:54.298635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.297841 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:54.298635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.298101 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:54.298635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.298385 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:54.319588 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.319545 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 23:06:54.329884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.320957 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:54.625935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.608951 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:54.626225 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.626187 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:54.648878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.645081 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:54.648878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.645164 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:54.751889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.750935 5778 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:54.786878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.781036 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" Nov 01 23:06:54.826925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.815504 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96" event={"ID":"a58fae54-0661-4c07-9c3e-3e4329b24def","Type":"ContainerDied","Data":"9a3d668f5ba50bf50f8ab3fa3e5c612cdbf043b2cfd357cb503ded6546bda89f"} Nov 01 23:06:54.826925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.815575 5778 scope.go:122] "RemoveContainer" containerID="7b466934cb2227de208a95dcf43aacb940f3596ced24895908aafc99096ac73b" Nov 01 23:06:54.826925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.819943 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:54.826925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.819992 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:54.968893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.949188 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:54.968893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:54.949795 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:06:55.376188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.376133 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:55.404736 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.381668 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 23:06:55.447911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.447807 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:06:55.470425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.470394 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 23:06:55.490880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.485818 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 23:06:55.491148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.491117 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 23:06:55.491696 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.491664 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:55.496360 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.495305 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:55.601306 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.599283 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:55.601306 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.599347 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:55.603510 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.603487 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:06:55.608474 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.608435 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:06:55.680825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:55.674933 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fb5b4-85554bdc66-t9v96"] Nov 01 23:06:56.147147 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.147071 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 23:06:56.184671 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.177039 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 23:06:56.311643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.311602 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:56.311948 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.311919 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:56.312139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.312119 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:06:56.372142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.372102 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:06:56.396696 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.396665 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" path="/var/lib/kubelet/pods/a58fae54-0661-4c07-9c3e-3e4329b24def/volumes" Nov 01 23:06:56.472329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.471116 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:06:56.481408 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.478389 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:06:56.587881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.587693 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:56.587881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:56.587759 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:57.271771 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.270291 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:06:57.272949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.272342 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:06:57.273630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.273563 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:06:57.387683 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.387639 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" event={"ID":"8fdd665f-021d-4fa4-a753-d0051b31c6d8","Type":"ContainerStarted","Data":"effc8aa9915de506545562b7e24e36c1b83fdf7007d5a1bb5a9da41cfcfec5d5"} Nov 01 23:06:57.387901 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.387699 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" event={"ID":"8fdd665f-021d-4fa4-a753-d0051b31c6d8","Type":"ContainerStarted","Data":"426b16f5d795ec8d28990420ddc17b0473c437294e511f4cf16eb49525d46aa1"} Nov 01 23:06:57.445911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.413421 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podStartSLOduration=5.413405546 podStartE2EDuration="5.413405546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:57.411594736 +0000 UTC m=+2587.187581598" watchObservedRunningTime="2026-11-01 23:06:57.413405546 +0000 UTC m=+2587.189392464" Nov 01 23:06:57.588356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.588252 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:57.588356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:57.588310 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:58.303932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:58.302698 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:06:58.304582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:58.304553 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:06:58.304816 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:58.304794 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:06:58.587444 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:58.587336 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:58.588026 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:58.587967 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:06:59.308015 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:59.307979 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:06:59.308309 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:59.308281 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:06:59.588193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:59.588105 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:06:59.588193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:06:59.588170 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 23:07:00.302937 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:00.302894 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:07:00.303157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:00.302955 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:07:00.592619 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:00.592525 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:07:00.593087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:00.593031 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:07:00.593805 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:00.593784 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:07:01.302664 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:01.302634 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:07:01.302862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:01.302690 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:07:02.302371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:02.302327 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:07:02.302897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:02.302391 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:07:03.302868 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:03.302814 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:07:03.303408 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:03.302900 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:07:03.761615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:03.761582 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:07:03.773382 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:03.773350 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:07:04.302457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:04.302418 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:07:04.302651 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:04.302488 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 23:07:05.307586 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:05.307557 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:07:05.308104 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:05.308088 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:07:05.308794 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:05.308775 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:07:13.184232 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.184174 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:07:13.186436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.186149 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerName="maas-api" Nov 01 23:07:13.186622 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.186603 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerName="maas-api" Nov 01 23:07:13.187027 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.187007 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58fae54-0661-4c07-9c3e-3e4329b24def" containerName="maas-api" Nov 01 23:07:13.218527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.218490 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:07:13.218915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.218892 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.228323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.228301 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9d2a4e-serving-cert\"" Nov 01 23:07:13.351950 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.351903 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.357359 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.357325 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.362396 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.357567 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.378996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.376839 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g869p\" (UniqueName: \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.480232 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.480131 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g869p\" (UniqueName: \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.480441 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.480420 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.480550 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.480526 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.480617 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.480572 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.481640 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.481609 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.487040 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.487012 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.488887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.488818 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.490943 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.490838 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g869p\" (UniqueName: \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\") pod \"maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.550193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.550158 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:13.867056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:13.867025 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:07:14.720982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:14.720936 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" event={"ID":"335b477e-bf6c-42e4-b7fc-ab9628fa7094","Type":"ContainerStarted","Data":"ede54bd91ca326dd39f7407631a58f1b6cb310cf2d73844acb39b559714d2e38"} Nov 01 23:07:14.721506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:14.721006 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" event={"ID":"335b477e-bf6c-42e4-b7fc-ab9628fa7094","Type":"ContainerStarted","Data":"fc084426c31142df953afe3452be137d0915d0128c7b24cc1b257ee27cb05bb9"} Nov 01 23:07:14.721506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:14.721064 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:14.741650 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:14.741596 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" podStartSLOduration=1.244434553 podStartE2EDuration="1.74158269s" totalImagesPullingTime="497.148137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:14.738109363 +0000 UTC m=+2604.514096193" watchObservedRunningTime="2026-11-01 23:07:14.74158269 +0000 UTC m=+2604.517569519" Nov 01 23:07:18.779147 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:07:18.779106 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:07:19.709827 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:19.709790 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:07:19.710080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:19.710057 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" containerID="cri-o://2dc95e1f5895ed2f889fb9d91d57250301a7b1597f659ca5966729c7bbfedc37" gracePeriod=30 Nov 01 23:07:20.735107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:20.735081 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:07:24.853685 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:24.853651 5778 generic.go:356] "Generic (PLEG): container finished" podID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerID="2dc95e1f5895ed2f889fb9d91d57250301a7b1597f659ca5966729c7bbfedc37" exitCode=0 Nov 01 23:07:24.854131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:24.853729 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerDied","Data":"2dc95e1f5895ed2f889fb9d91d57250301a7b1597f659ca5966729c7bbfedc37"} Nov 01 23:07:24.854131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:24.853827 5778 scope.go:122] "RemoveContainer" containerID="4ef042699db460dcb3bef641267fcdde94571145a2ea38350172454bd2dbc715" Nov 01 23:07:25.021715 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.021682 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:07:25.111024 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.110984 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111068 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111130 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111167 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111229 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111301 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111371 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111294 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:25.111445 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111391 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:25.111544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111470 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111524 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\") pod \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\" (UID: \"3ff65332-db64-4bfe-8d8e-3076dc3f4a98\") " Nov 01 23:07:25.111663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111626 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:25.111876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.111772 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:25.112127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112004 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:25.112496 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112471 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.112594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112501 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.112594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112516 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.112594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112530 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.112594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.112544 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.113791 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.113766 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:25.113962 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.113863 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:25.114198 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.114171 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:07:25.114294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.114206 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb" pod "3ff65332-db64-4bfe-8d8e-3076dc3f4a98" (UID: "3ff65332-db64-4bfe-8d8e-3076dc3f4a98"). InnerVolumeSpecName "kube-api-access-4vrbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:25.189556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.189513 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 23:07:25.214746 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.214710 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrbb\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-kube-api-access-4vrbb\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.214915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.214750 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.214915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.214766 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.214915 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.214780 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff65332-db64-4bfe-8d8e-3076dc3f4a98-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:25.567347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.567309 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:07:25.881056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.880952 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" event={"ID":"3ff65332-db64-4bfe-8d8e-3076dc3f4a98","Type":"ContainerDied","Data":"4e3b05af559b667cfd4dc8ba4f7684eab547f9c21cf7bb3e948d1b462626ad16"} Nov 01 23:07:25.881056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.881023 5778 scope.go:122] "RemoveContainer" containerID="2dc95e1f5895ed2f889fb9d91d57250301a7b1597f659ca5966729c7bbfedc37" Nov 01 23:07:25.881628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.881190 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf" Nov 01 23:07:25.931029 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.930997 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:07:25.948143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:25.948111 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-85b1ad-openshift-default-7b8f4696c-b84xf"] Nov 01 23:07:26.375774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:26.375743 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" path="/var/lib/kubelet/pods/3ff65332-db64-4bfe-8d8e-3076dc3f4a98/volumes" Nov 01 23:07:29.833371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:29.833139 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:07:40.216712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.216666 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" containerID="cri-o://9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b" gracePeriod=45 Nov 01 23:07:40.526254 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.526223 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:07:40.598182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.598127 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" containerID="cri-o://a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25" gracePeriod=45 Nov 01 23:07:40.616044 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616017 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\" (UniqueName: \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616194 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616134 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616263 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616252 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616288 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616406 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616339 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616489 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616433 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\") pod \"3e6d3ca2-80af-4215-b87a-22970a3409d7\" (UID: \"3e6d3ca2-80af-4215-b87a-22970a3409d7\") " Nov 01 23:07:40.616608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.616571 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.617107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.617086 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.617258 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.617231 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.617320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.617305 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.620164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.620129 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.621272 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.621244 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:40.623812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.623754 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb" pod "3e6d3ca2-80af-4215-b87a-22970a3409d7" (UID: "3e6d3ca2-80af-4215-b87a-22970a3409d7"). InnerVolumeSpecName "kube-api-access-zhknb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:40.668157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.664973 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": dial tcp 10.132.0.173:8000: connect: connection refused" Nov 01 23:07:40.718739 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.718707 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.718900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.718743 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.718900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.718758 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6d3ca2-80af-4215-b87a-22970a3409d7-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.718900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.718772 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhknb\" (UniqueName: \"kubernetes.io/projected/3e6d3ca2-80af-4215-b87a-22970a3409d7-kube-api-access-zhknb\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.718900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.718788 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3ca2-80af-4215-b87a-22970a3409d7-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.928741 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:40.928715 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:07:41.034449 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034419 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\" (UniqueName: \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034642 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034506 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034642 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034600 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034651 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034710 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034768 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\") pod \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\" (UID: \"fffa1b57-d0a2-41a5-b91e-691495c31a8f\") " Nov 01 23:07:41.034967 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.034763 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:41.035282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.035255 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:41.035382 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.035274 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:41.035552 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.035529 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.035630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.035554 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.035630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.035570 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.037739 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.037710 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "kube-api-access-n5qlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:41.037953 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.037926 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:41.038020 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.037963 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm" pod "fffa1b57-d0a2-41a5-b91e-691495c31a8f" (UID: "fffa1b57-d0a2-41a5-b91e-691495c31a8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:41.137224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.137151 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qlt\" (UniqueName: \"kubernetes.io/projected/fffa1b57-d0a2-41a5-b91e-691495c31a8f-kube-api-access-n5qlt\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.137224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.137183 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffa1b57-d0a2-41a5-b91e-691495c31a8f-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.137224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.137196 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffa1b57-d0a2-41a5-b91e-691495c31a8f-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:41.172931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.172899 5778 generic.go:356] "Generic (PLEG): container finished" podID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerID="9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b" exitCode=0 Nov 01 23:07:41.173065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.172947 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" Nov 01 23:07:41.173065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.172981 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" event={"ID":"3e6d3ca2-80af-4215-b87a-22970a3409d7","Type":"ContainerDied","Data":"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b"} Nov 01 23:07:41.173065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.173029 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc" event={"ID":"3e6d3ca2-80af-4215-b87a-22970a3409d7","Type":"ContainerDied","Data":"572991aa19124cc3445c4fd44523a2f5d8d9f2ddfda2f948a029ca3344b1da8a"} Nov 01 23:07:41.173065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.173053 5778 scope.go:122] "RemoveContainer" containerID="9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b" Nov 01 23:07:41.174565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.174536 5778 generic.go:356] "Generic (PLEG): container finished" podID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerID="a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25" exitCode=0 Nov 01 23:07:41.174690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.174574 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" Nov 01 23:07:41.174690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.174571 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" event={"ID":"fffa1b57-d0a2-41a5-b91e-691495c31a8f","Type":"ContainerDied","Data":"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25"} Nov 01 23:07:41.174690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.174616 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb" event={"ID":"fffa1b57-d0a2-41a5-b91e-691495c31a8f","Type":"ContainerDied","Data":"e61126a38328d892b473d72e0c1965f69daa9d296e307c32a0cadde8ec4b1659"} Nov 01 23:07:41.202406 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.202367 5778 scope.go:122] "RemoveContainer" containerID="9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b" Nov 01 23:07:41.202834 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:07:41.202808 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b\": container with ID starting with 9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b not found: ID does not exist" containerID="9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b" Nov 01 23:07:41.203385 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.202864 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b"} err="failed to get container status \"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b\": rpc error: code = NotFound desc = could not find container \"9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b\": container with ID starting with 9f627970fefa3fd6b2eef508996787210bc9e58744a473878e42589c9debee4b not found: ID does not exist" Nov 01 23:07:41.203385 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.202889 5778 scope.go:122] "RemoveContainer" containerID="a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25" Nov 01 23:07:41.206977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.206954 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 23:07:41.210506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.210484 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/premium-w4-2f3705-kserve-7cbcbfbc57-6x2gc"] Nov 01 23:07:41.213488 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.213466 5778 scope.go:122] "RemoveContainer" containerID="a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25" Nov 01 23:07:41.213934 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:07:41.213901 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25\": container with ID starting with a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25 not found: ID does not exist" containerID="a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25" Nov 01 23:07:41.214040 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.213941 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25"} err="failed to get container status \"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25\": rpc error: code = NotFound desc = could not find container \"a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25\": container with ID starting with a87d4304ccebd182362065b6ce3e2aa076291e15685e2fec8129f454dfeacd25 not found: ID does not exist" Nov 01 23:07:41.217524 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.217499 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:07:41.222798 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.222771 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/sub-test-model-ee07ad-kserve-74d95f645c-62krb"] Nov 01 23:07:41.783473 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.783287 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:07:41.783681 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:41.783547 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" containerID="cri-o://c2100bbf253eff6a116ba785acd84835f91bfb34560e2842d1821832981f00ab" gracePeriod=30 Nov 01 23:07:42.375346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.375308 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" path="/var/lib/kubelet/pods/3e6d3ca2-80af-4215-b87a-22970a3409d7/volumes" Nov 01 23:07:42.375817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.375801 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" path="/var/lib/kubelet/pods/fffa1b57-d0a2-41a5-b91e-691495c31a8f/volumes" Nov 01 23:07:42.660007 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.659972 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:07:42.661016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.660990 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" Nov 01 23:07:42.661016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661014 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661062 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661070 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661113 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661121 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661139 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661145 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661302 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661319 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff65332-db64-4bfe-8d8e-3076dc3f4a98" containerName="istio-proxy" Nov 01 23:07:42.661409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661333 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffa1b57-d0a2-41a5-b91e-691495c31a8f" containerName="main" Nov 01 23:07:42.661409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.661349 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6d3ca2-80af-4215-b87a-22970a3409d7" containerName="main" Nov 01 23:07:42.667202 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.667159 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.684217 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.678799 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"openshift-service-ca.crt\"" Nov 01 23:07:42.684217 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.679437 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:07:42.684217 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.679608 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"kube-root-ca.crt\"" Nov 01 23:07:42.753091 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.753057 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"default-dockercfg-c9cjm\"" Nov 01 23:07:42.753091 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.753060 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee07ad\"/\"rate-test-model-ee07ad-kserve-self-signed-certs\"" Nov 01 23:07:42.783171 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783132 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.783349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783222 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.783403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783366 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cw2w\" (UniqueName: \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.783452 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783418 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.783580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783556 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.783700 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.783676 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885503 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885464 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cw2w\" (UniqueName: \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885527 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885579 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885621 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885692 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.885973 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.885726 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.886058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.886032 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.886136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.886066 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.886203 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.886150 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.888480 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.888450 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.888678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.888656 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.935921 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.931542 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cw2w\" (UniqueName: \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\") pod \"rate-test-model-ee07ad-kserve-8476575b8f-ndvjj\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:42.988204 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:42.988140 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:43.390271 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:43.390242 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:07:43.392631 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:07:43.392606 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b12473d_3d7d_46fb_ae16_569af651e5da.slice/crio-758349a4602f1cec7f790ee3d30a8d856730399bb0d713ba49339d3f36d5e80e WatchSource:0}: Error finding container 758349a4602f1cec7f790ee3d30a8d856730399bb0d713ba49339d3f36d5e80e: Status 404 returned error can't find the container with id 758349a4602f1cec7f790ee3d30a8d856730399bb0d713ba49339d3f36d5e80e Nov 01 23:07:43.629208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:43.629169 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:07:43.629384 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:43.629238 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:07:44.211898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.211818 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" event={"ID":"5b12473d-3d7d-46fb-ae16-569af651e5da","Type":"ContainerStarted","Data":"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738"} Nov 01 23:07:44.211898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.211882 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" event={"ID":"5b12473d-3d7d-46fb-ae16-569af651e5da","Type":"ContainerStarted","Data":"758349a4602f1cec7f790ee3d30a8d856730399bb0d713ba49339d3f36d5e80e"} Nov 01 23:07:44.233838 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.233761 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" podStartSLOduration=2.2337434959999998 podStartE2EDuration="2.233743496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:44.228556048 +0000 UTC m=+2634.004542877" watchObservedRunningTime="2026-11-01 23:07:44.233743496 +0000 UTC m=+2634.009730325" Nov 01 23:07:44.879436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.879398 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" containerID="cri-o://7ac5b81397ca65b4590304404e8fea9d03e2ace5a2337718bf5314d20fa4385c" gracePeriod=45 Nov 01 23:07:44.976302 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.976261 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": dial tcp 10.132.0.174:8000: connect: connection refused" Nov 01 23:07:44.989019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:44.988991 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:45.224790 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.224743 5778 generic.go:356] "Generic (PLEG): container finished" podID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerID="7ac5b81397ca65b4590304404e8fea9d03e2ace5a2337718bf5314d20fa4385c" exitCode=0 Nov 01 23:07:45.228307 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.225165 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" event={"ID":"fb7a701a-a262-46b4-aab6-13c4c64f303b","Type":"ContainerDied","Data":"7ac5b81397ca65b4590304404e8fea9d03e2ace5a2337718bf5314d20fa4385c"} Nov 01 23:07:45.253235 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.253209 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:07:45.340300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.340266 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.340500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.340407 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.340841 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.340819 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.341150 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341115 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:45.341298 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341275 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.341374 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341359 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.341469 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341454 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\" (UniqueName: \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\") pod \"fb7a701a-a262-46b4-aab6-13c4c64f303b\" (UID: \"fb7a701a-a262-46b4-aab6-13c4c64f303b\") " Nov 01 23:07:45.341528 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341504 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:45.342687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.341695 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:45.342687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.342259 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.342687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.342283 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.342687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.342299 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.344145 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.344121 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:45.344145 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.344126 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:45.344888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.344831 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm" pod "fb7a701a-a262-46b4-aab6-13c4c64f303b" (UID: "fb7a701a-a262-46b4-aab6-13c4c64f303b"). InnerVolumeSpecName "kube-api-access-927jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:45.443574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.443544 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a701a-a262-46b4-aab6-13c4c64f303b-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.443745 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.443731 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a701a-a262-46b4-aab6-13c4c64f303b-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.443873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:45.443836 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927jm\" (UniqueName: \"kubernetes.io/projected/fb7a701a-a262-46b4-aab6-13c4c64f303b-kube-api-access-927jm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:46.237491 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.237452 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" Nov 01 23:07:46.237491 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.237468 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v" event={"ID":"fb7a701a-a262-46b4-aab6-13c4c64f303b","Type":"ContainerDied","Data":"a0f6270e085980b8a906bc5a5a7764c98be339bf770177acbe1b7c1934c3d518"} Nov 01 23:07:46.238044 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.237530 5778 scope.go:122] "RemoveContainer" containerID="7ac5b81397ca65b4590304404e8fea9d03e2ace5a2337718bf5314d20fa4385c" Nov 01 23:07:46.266619 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.266587 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:07:46.270001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.269976 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/sub-test-model-dc635e-kserve-668c7fc89-j757v"] Nov 01 23:07:46.377265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:46.377232 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" path="/var/lib/kubelet/pods/fb7a701a-a262-46b4-aab6-13c4c64f303b/volumes" Nov 01 23:07:47.250285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.250196 5778 generic.go:356] "Generic (PLEG): container finished" podID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerID="c2100bbf253eff6a116ba785acd84835f91bfb34560e2842d1821832981f00ab" exitCode=0 Nov 01 23:07:47.250801 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.250295 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerDied","Data":"c2100bbf253eff6a116ba785acd84835f91bfb34560e2842d1821832981f00ab"} Nov 01 23:07:47.250801 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.250353 5778 scope.go:122] "RemoveContainer" containerID="b7ea7232081c585ccd4bdbcb5e08d6fd994bae7d05a3ed2e55fbc5392e0f7dd8" Nov 01 23:07:47.285330 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.285299 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:07:47.378190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378010 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378096 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378127 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378182 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378257 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378306 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378350 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378377 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.378551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.378413 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\") pod \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\" (UID: \"d28cceb0-fcd1-4831-bfb6-55af50b323b0\") " Nov 01 23:07:47.379391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.379356 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:47.387368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.380490 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:47.387368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.380949 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:47.387368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.381251 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:47.387368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.381877 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:47.399527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.399494 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:47.399674 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.399649 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:07:47.400276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.400246 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "kube-api-access-4bqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:47.401320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.401291 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token" pod "d28cceb0-fcd1-4831-bfb6-55af50b323b0" (UID: "d28cceb0-fcd1-4831-bfb6-55af50b323b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:47.480792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480756 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.480792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480790 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480801 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480809 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480818 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480826 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480833 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480842 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqhx\" (UniqueName: \"kubernetes.io/projected/d28cceb0-fcd1-4831-bfb6-55af50b323b0-kube-api-access-4bqhx\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:47.481011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:47.480899 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28cceb0-fcd1-4831-bfb6-55af50b323b0-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:48.260609 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.260562 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" event={"ID":"d28cceb0-fcd1-4831-bfb6-55af50b323b0","Type":"ContainerDied","Data":"4d71f4bd34aa38927bc22f6fdd0f8625d26da2703fd9baa03e74e75562f9b9dc"} Nov 01 23:07:48.260609 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.260595 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr" Nov 01 23:07:48.261219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.260642 5778 scope.go:122] "RemoveContainer" containerID="c2100bbf253eff6a116ba785acd84835f91bfb34560e2842d1821832981f00ab" Nov 01 23:07:48.295694 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.295665 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:07:48.298941 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.298904 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3381-openshift-default-b44b757cf-k26lr"] Nov 01 23:07:48.378202 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:48.378171 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" path="/var/lib/kubelet/pods/d28cceb0-fcd1-4831-bfb6-55af50b323b0/volumes" Nov 01 23:07:48.777039 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:07:48.777004 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:07:50.214897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.214820 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:07:50.215818 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.215787 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:07:50.215818 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.215818 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:07:50.216065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.215915 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" Nov 01 23:07:50.216065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.215925 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" Nov 01 23:07:50.216193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.216141 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:07:50.216193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.216166 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:07:50.216193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.216178 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7a701a-a262-46b4-aab6-13c4c64f303b" containerName="main" Nov 01 23:07:50.229175 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.229150 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.247709 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.247681 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-01e30d-openshift-default-dockercfg-vsndd\"" Nov 01 23:07:50.253017 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.252965 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:07:50.338642 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.338598 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.338642 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.338643 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.338929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.338772 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.338929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.338836 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.338929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.338923 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.339100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.339024 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.339184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.339163 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j6m\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.339250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.339205 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.339250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.339227 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.474931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.474755 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j6m\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.474931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.474833 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.474931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.474888 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.474931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.474912 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.495258 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.495389 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.495432 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.495498 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.495635 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.497428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.496670 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.514822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.513676 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.515482 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.515174 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.521876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.515908 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.522378 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.522351 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.587060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.587026 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.589932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.589836 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.590796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.590770 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j6m\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.614544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.614491 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:50.858006 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.857905 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-01e30d-openshift-default-dockercfg-vsndd\"" Nov 01 23:07:50.863074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:50.863045 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:51.174597 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.174563 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:07:51.179461 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:07:51.179433 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405525e1_ba54_4275_b2f1_7c31e8f88f82.slice/crio-8dd2dd7ec4d8a40e5efb832772a630bf2687604adbbc4ead114853f9f7fe5a61 WatchSource:0}: Error finding container 8dd2dd7ec4d8a40e5efb832772a630bf2687604adbbc4ead114853f9f7fe5a61: Status 404 returned error can't find the container with id 8dd2dd7ec4d8a40e5efb832772a630bf2687604adbbc4ead114853f9f7fe5a61 Nov 01 23:07:51.182701 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.182665 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:07:51.182825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.182739 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:07:51.182825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.182774 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:07:51.293095 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.293063 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerStarted","Data":"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9"} Nov 01 23:07:51.293640 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.293613 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerStarted","Data":"8dd2dd7ec4d8a40e5efb832772a630bf2687604adbbc4ead114853f9f7fe5a61"} Nov 01 23:07:51.316956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.316900 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podStartSLOduration=1.316880558 podStartE2EDuration="1.316880558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:51.312367256 +0000 UTC m=+2641.088354087" watchObservedRunningTime="2026-11-01 23:07:51.316880558 +0000 UTC m=+2641.092867385" Nov 01 23:07:51.863638 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:51.863602 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:52.870922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:52.870872 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:52.989602 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:52.989566 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:53.002845 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:53.002819 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:07:53.319020 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:53.318992 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:53.321424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:53.321345 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:07:53.336127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:07:53.336101 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:08:02.341869 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.341817 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:08:02.345873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.342778 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:08:02.345873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.342806 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28cceb0-fcd1-4831-bfb6-55af50b323b0" containerName="istio-proxy" Nov 01 23:08:02.359960 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.358666 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.390489 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.390307 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:08:02.512202 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512167 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.512535 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512501 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.512691 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512653 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.512772 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512710 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlbj\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.512955 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512930 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.513030 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.512981 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.513095 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.513070 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.513199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.513173 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.513268 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.513216 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614467 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614368 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614467 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614431 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614469 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614549 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614603 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614710 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614806 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.614920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614872 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.615091 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.614929 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlbj\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.615291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.615266 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.615624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.615601 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.616036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.615985 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.616509 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.616485 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.616963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.616942 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.620931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.620879 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.623472 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.623449 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.625139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.625117 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.627137 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.626359 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlbj\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:02.705454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:02.705158 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:03.062669 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:08:03.062633 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1759788_2268_4a5d_9c80_ec6a2b39d528.slice/crio-c366199d50fcc5f89888d07bf24ec4ca86fd5dc18ba5cf754c2bb83f127e766b WatchSource:0}: Error finding container c366199d50fcc5f89888d07bf24ec4ca86fd5dc18ba5cf754c2bb83f127e766b: Status 404 returned error can't find the container with id c366199d50fcc5f89888d07bf24ec4ca86fd5dc18ba5cf754c2bb83f127e766b Nov 01 23:08:03.065761 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.065724 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:03.065930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.065793 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:03.065930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.065822 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:03.066819 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.066791 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:08:03.419392 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.419355 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" event={"ID":"f1759788-2268-4a5d-9c80-ec6a2b39d528","Type":"ContainerStarted","Data":"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd"} Nov 01 23:08:03.420074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.419423 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" event={"ID":"f1759788-2268-4a5d-9c80-ec6a2b39d528","Type":"ContainerStarted","Data":"c366199d50fcc5f89888d07bf24ec4ca86fd5dc18ba5cf754c2bb83f127e766b"} Nov 01 23:08:03.446404 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.446334 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podStartSLOduration=1.446316529 podStartE2EDuration="1.446316529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:03.443167627 +0000 UTC m=+2653.219154461" watchObservedRunningTime="2026-11-01 23:08:03.446316529 +0000 UTC m=+2653.222303359" Nov 01 23:08:03.722572 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:03.719519 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:04.385409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.385377 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:08:04.395930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.395903 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:08:04.396222 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.396204 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.402869 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.402811 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"openshift-service-ca.crt\"" Nov 01 23:08:04.403154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.403137 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"rate-test-model-dc635e-kserve-self-signed-certs\"" Nov 01 23:08:04.404285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.404107 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"kube-root-ca.crt\"" Nov 01 23:08:04.407714 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.407322 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dc635e\"/\"default-dockercfg-hxt2g\"" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.580824 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.580974 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqq2c\" (UniqueName: \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.581023 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.581117 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.581153 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.585872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.581317 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.686980 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.687132 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.687235 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqq2c\" (UniqueName: \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.687265 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.687320 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.687360 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.688884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.688147 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.704878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.704285 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.704878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.704534 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.721935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.721901 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:04.722168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.722142 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:04.739790 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.739746 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.741891 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.741601 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:04.748712 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:04.748690 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqq2c\" (UniqueName: \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\") pod \"rate-test-model-dc635e-kserve-64586cc89c-ww956\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:05.024192 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:05.024100 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:05.349928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:05.347397 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:08:05.352121 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:08:05.352087 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cc550de_06fb_4a8f_817b_407ee81dcdaf.slice/crio-1b1aa037d057b8f2d6b912f1b0ed41645bdca2aaf95c1701678631e178f7032c WatchSource:0}: Error finding container 1b1aa037d057b8f2d6b912f1b0ed41645bdca2aaf95c1701678631e178f7032c: Status 404 returned error can't find the container with id 1b1aa037d057b8f2d6b912f1b0ed41645bdca2aaf95c1701678631e178f7032c Nov 01 23:08:05.522874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:05.522699 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" event={"ID":"5cc550de-06fb-4a8f-817b-407ee81dcdaf","Type":"ContainerStarted","Data":"1b1aa037d057b8f2d6b912f1b0ed41645bdca2aaf95c1701678631e178f7032c"} Nov 01 23:08:05.706308 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:05.706267 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:05.706692 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:05.706334 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:06.551191 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:06.551150 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" event={"ID":"5cc550de-06fb-4a8f-817b-407ee81dcdaf","Type":"ContainerStarted","Data":"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a"} Nov 01 23:08:06.577157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:06.577102 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" podStartSLOduration=2.577085989 podStartE2EDuration="2.577085989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:06.575187452 +0000 UTC m=+2656.351174295" watchObservedRunningTime="2026-11-01 23:08:06.577085989 +0000 UTC m=+2656.353072825" Nov 01 23:08:06.707876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:06.706268 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:06.707876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:06.706334 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:07.025876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:07.024922 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:07.706534 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:07.706503 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:07.706720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:07.706562 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:08.320876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:08.309588 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 23:08:08.320876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:08.309843 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" containerID="cri-o://af7b77d4f6f71f75d2da71c725fbc5160a61f89f2d9193078606e2fa0323ddb2" gracePeriod=30 Nov 01 23:08:08.707308 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:08.707263 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:08.707582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:08.707551 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:09.706265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:09.706225 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:08:09.706684 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:09.706294 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 23:08:10.711367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:10.711336 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:10.714668 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:10.711839 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:10.714668 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:10.712808 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:08:11.268978 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.268943 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:08:11.269303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.269190 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" podUID="e0199d0e-f878-46d9-a8c2-a5500284503b" containerName="maas-api" containerID="cri-o://823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04" gracePeriod=30 Nov 01 23:08:11.503440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.500688 5778 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:08:11.503440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.500745 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:08:11.637895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.637102 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:08:11.660876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.657001 5778 generic.go:356] "Generic (PLEG): container finished" podID="e0199d0e-f878-46d9-a8c2-a5500284503b" containerID="823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04" exitCode=0 Nov 01 23:08:11.660876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.657061 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" Nov 01 23:08:11.660876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.657083 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" event={"ID":"e0199d0e-f878-46d9-a8c2-a5500284503b","Type":"ContainerDied","Data":"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04"} Nov 01 23:08:11.660876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.657134 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5" event={"ID":"e0199d0e-f878-46d9-a8c2-a5500284503b","Type":"ContainerDied","Data":"1a371e1c6078ccd0c2f8398d5fc2c2dfde48f67d704331f3c5ca50d87aed7241"} Nov 01 23:08:11.660876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.657163 5778 scope.go:122] "RemoveContainer" containerID="823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04" Nov 01 23:08:11.672632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.672561 5778 scope.go:122] "RemoveContainer" containerID="823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04" Nov 01 23:08:11.672930 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:08:11.672903 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04\": container with ID starting with 823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04 not found: ID does not exist" containerID="823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04" Nov 01 23:08:11.673032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.672940 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04"} err="failed to get container status \"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04\": rpc error: code = NotFound desc = could not find container \"823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04\": container with ID starting with 823b0bae4cee1d9e9eb665aef9ff3c443ad86763d4769e2b01bcb92cf101fe04 not found: ID does not exist" Nov 01 23:08:11.712903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.710910 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\") pod \"e0199d0e-f878-46d9-a8c2-a5500284503b\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " Nov 01 23:08:11.712903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.710972 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\") pod \"e0199d0e-f878-46d9-a8c2-a5500284503b\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " Nov 01 23:08:11.712903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.711010 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\") pod \"e0199d0e-f878-46d9-a8c2-a5500284503b\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " Nov 01 23:08:11.712903 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.712057 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\" (UniqueName: \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\") pod \"e0199d0e-f878-46d9-a8c2-a5500284503b\" (UID: \"e0199d0e-f878-46d9-a8c2-a5500284503b\") " Nov 01 23:08:11.717894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.715455 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca" pod "e0199d0e-f878-46d9-a8c2-a5500284503b" (UID: "e0199d0e-f878-46d9-a8c2-a5500284503b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:11.722898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.720983 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls" pod "e0199d0e-f878-46d9-a8c2-a5500284503b" (UID: "e0199d0e-f878-46d9-a8c2-a5500284503b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:11.722898 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.721098 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd" pod "e0199d0e-f878-46d9-a8c2-a5500284503b" (UID: "e0199d0e-f878-46d9-a8c2-a5500284503b"). InnerVolumeSpecName "kube-api-access-lc5wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:11.750236 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.750090 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs" pod "e0199d0e-f878-46d9-a8c2-a5500284503b" (UID: "e0199d0e-f878-46d9-a8c2-a5500284503b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:11.929879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.926866 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.929879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.926901 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0199d0e-f878-46d9-a8c2-a5500284503b-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.929879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.926919 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0199d0e-f878-46d9-a8c2-a5500284503b-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.929879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:11.926932 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5wd\" (UniqueName: \"kubernetes.io/projected/e0199d0e-f878-46d9-a8c2-a5500284503b-kube-api-access-lc5wd\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:12.082622 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:12.082585 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:08:12.086399 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:12.086363 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-00185d-69c9dc48bc-jlzx5"] Nov 01 23:08:12.377555 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:12.377460 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0199d0e-f878-46d9-a8c2-a5500284503b" path="/var/lib/kubelet/pods/e0199d0e-f878-46d9-a8c2-a5500284503b/volumes" Nov 01 23:08:13.677970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.677943 5778 generic.go:356] "Generic (PLEG): container finished" podID="afe9f559-e843-40c3-901d-fde2eba8250b" containerID="af7b77d4f6f71f75d2da71c725fbc5160a61f89f2d9193078606e2fa0323ddb2" exitCode=0 Nov 01 23:08:13.678368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.678031 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" event={"ID":"afe9f559-e843-40c3-901d-fde2eba8250b","Type":"ContainerDied","Data":"af7b77d4f6f71f75d2da71c725fbc5160a61f89f2d9193078606e2fa0323ddb2"} Nov 01 23:08:13.678368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.678112 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" event={"ID":"afe9f559-e843-40c3-901d-fde2eba8250b","Type":"ContainerDied","Data":"69e28b000ec9649046ab6027159b9d84bce36125e762d9426bc0757bdaca4c7c"} Nov 01 23:08:13.678368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.678133 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e28b000ec9649046ab6027159b9d84bce36125e762d9426bc0757bdaca4c7c" Nov 01 23:08:13.679492 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.679474 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 23:08:13.793954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.793922 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.793982 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794049 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794100 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794130 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794190 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794226 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.794315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794251 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794441 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794369 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.794478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794430 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.794513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794489 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.794546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794530 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:13.794616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794596 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.794681 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.794646 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\") pod \"afe9f559-e843-40c3-901d-fde2eba8250b\" (UID: \"afe9f559-e843-40c3-901d-fde2eba8250b\") " Nov 01 23:08:13.795221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.795197 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.795221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.795218 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.795221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.795228 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe9f559-e843-40c3-901d-fde2eba8250b-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.795493 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.795237 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.795493 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.795245 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.796515 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.796492 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.796767 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.796749 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "kube-api-access-s4vjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:13.796891 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.796819 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:08:13.797156 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.797138 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token" pod "afe9f559-e843-40c3-901d-fde2eba8250b" (UID: "afe9f559-e843-40c3-901d-fde2eba8250b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:13.896417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.896350 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4vjg\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-kube-api-access-s4vjg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.896417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.896378 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe9f559-e843-40c3-901d-fde2eba8250b-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.896417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.896389 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe9f559-e843-40c3-901d-fde2eba8250b-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.896417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:13.896398 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe9f559-e843-40c3-901d-fde2eba8250b-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:14.685552 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:14.685522 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb" Nov 01 23:08:14.707414 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:14.707388 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 23:08:14.714071 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:14.714045 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-khwcb"] Nov 01 23:08:15.024974 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:15.024881 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:15.037704 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:15.037671 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:15.758302 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:15.758266 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:08:16.376019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:16.375986 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" path="/var/lib/kubelet/pods/afe9f559-e843-40c3-901d-fde2eba8250b/volumes" Nov 01 23:08:18.597448 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:18.597406 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:08:18.597874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:18.597659 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" containerID="cri-o://b89b30b98998c294b84ea516bd2e2f1502e4d7db0de8d20a2b0acf6d19069ba5" gracePeriod=30 Nov 01 23:08:18.933605 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:08:18.933470 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:08:23.875721 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:23.875628 5778 generic.go:356] "Generic (PLEG): container finished" podID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerID="b89b30b98998c294b84ea516bd2e2f1502e4d7db0de8d20a2b0acf6d19069ba5" exitCode=0 Nov 01 23:08:23.876248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:23.875744 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerDied","Data":"b89b30b98998c294b84ea516bd2e2f1502e4d7db0de8d20a2b0acf6d19069ba5"} Nov 01 23:08:23.876248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:23.875824 5778 scope.go:122] "RemoveContainer" containerID="bd275253900f9a5311ec3e4ea3eaea18f136f912e717c90cd114627bae5117ed" Nov 01 23:08:23.911691 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:23.911670 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:08:24.087182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087151 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087226 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087301 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087326 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087377 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087405 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087468 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:24.087558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087484 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:24.087558 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087535 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087624 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087677 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:24.087834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087691 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\") pod \"d4eab4d2-d13d-4d44-9331-93de4cb81141\" (UID: \"d4eab4d2-d13d-4d44-9331-93de4cb81141\") " Nov 01 23:08:24.087834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087693 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:24.087834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.087743 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:24.088454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.088358 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eab4d2-d13d-4d44-9331-93de4cb81141-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.088454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.088382 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.088454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.088392 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.088454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.088401 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.088454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.088409 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.090166 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.090047 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:24.090464 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.090435 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "kube-api-access-pf2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:24.090464 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.090440 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:08:24.090596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.090476 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token" pod "d4eab4d2-d13d-4d44-9331-93de4cb81141" (UID: "d4eab4d2-d13d-4d44-9331-93de4cb81141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:24.189731 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.189705 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.189731 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.189731 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.189954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.189741 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eab4d2-d13d-4d44-9331-93de4cb81141-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.189954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.189750 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2kq\" (UniqueName: \"kubernetes.io/projected/d4eab4d2-d13d-4d44-9331-93de4cb81141-kube-api-access-pf2kq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:24.894878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.890782 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" Nov 01 23:08:24.894878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.891008 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz" event={"ID":"d4eab4d2-d13d-4d44-9331-93de4cb81141","Type":"ContainerDied","Data":"5d677637902af3fe1790a7a168cba49b507a4e31e0687c4de1c5d5b29f7d13f8"} Nov 01 23:08:24.894878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.891474 5778 scope.go:122] "RemoveContainer" containerID="b89b30b98998c294b84ea516bd2e2f1502e4d7db0de8d20a2b0acf6d19069ba5" Nov 01 23:08:24.932424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.931663 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:08:24.939883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:24.937251 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-00185d-openshift-default-7f7c4c9d89-jsllz"] Nov 01 23:08:26.403876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:26.403217 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" path="/var/lib/kubelet/pods/d4eab4d2-d13d-4d44-9331-93de4cb81141/volumes" Nov 01 23:08:34.288270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:34.288238 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv_e98a9dd7-b1fc-4a18-b316-78d47b0ebd87/payload-processing/0.log" Nov 01 23:08:35.274035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274005 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:08:35.274637 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274622 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:08:35.274705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274638 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:08:35.274705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274660 5778 cpu_manager.go:414] "containerMap: removing container" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" Nov 01 23:08:35.274705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274666 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" Nov 01 23:08:35.274705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274676 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e0199d0e-f878-46d9-a8c2-a5500284503b" containerName="maas-api" Nov 01 23:08:35.274705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274682 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0199d0e-f878-46d9-a8c2-a5500284503b" containerName="maas-api" Nov 01 23:08:35.274919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274813 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:08:35.274919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274830 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:08:35.274919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274843 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe9f559-e843-40c3-901d-fde2eba8250b" containerName="istio-proxy" Nov 01 23:08:35.274919 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.274869 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0199d0e-f878-46d9-a8c2-a5500284503b" containerName="maas-api" Nov 01 23:08:35.280031 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.280011 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.290044 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.290022 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d2a4e-openshift-default-dockercfg-6kq29\"" Nov 01 23:08:35.294953 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.294928 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:08:35.326446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326412 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326604 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326530 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326604 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326577 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326607 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326649 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326890 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326735 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326890 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326772 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.326890 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326872 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vng8l\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.327061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.326918 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428563 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428516 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428751 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428625 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vng8l\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428751 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428668 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428751 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428740 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428829 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428889 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428919 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.428982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428965 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.428984 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.429060 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.429107 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429348 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.429208 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429449 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.429423 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.429981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.429559 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.431623 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.431602 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.431946 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.431929 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.439546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.439520 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vng8l\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.441148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.441122 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.597744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.597668 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:35.959547 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:08:35.951812 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec5eae9_9d6c_4a46_82f7_70abf5eb9ec7.slice/crio-eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44 WatchSource:0}: Error finding container eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44: Status 404 returned error can't find the container with id eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44 Nov 01 23:08:35.959547 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.953969 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:08:35.967912 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.966967 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:35.967912 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.967066 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:35.967912 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:35.967111 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:08:36.002758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:36.002722 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" event={"ID":"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7","Type":"ContainerStarted","Data":"eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44"} Nov 01 23:08:37.013224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:37.013192 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" event={"ID":"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7","Type":"ContainerStarted","Data":"9781e802180009aa35d5e94fea685c1e4d5c0b35a6b01e8987842a10f7365f5d"} Nov 01 23:08:37.040802 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:37.040693 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podStartSLOduration=2.04067119 podStartE2EDuration="2.04067119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:37.037321356 +0000 UTC m=+2686.813308186" watchObservedRunningTime="2026-11-01 23:08:37.04067119 +0000 UTC m=+2686.816658019" Nov 01 23:08:37.600876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:37.600311 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:37.600876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:37.600439 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:37.600876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:37.600497 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:38.598730 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:38.598695 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:38.599147 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:38.598747 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:39.653894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:39.613522 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:08:39.653894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:39.642496 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" podUID="dde1c837-949d-4349-9861-2135cda48339" containerName="maas-api" containerID="cri-o://0ab8b069bfdc02a918cc85cecbc6f5589451fc559f58fc6a519ad9bc1829b1d7" gracePeriod=30 Nov 01 23:08:39.657809 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:39.657769 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:39.658128 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:39.658092 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:40.104625 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.104587 5778 generic.go:356] "Generic (PLEG): container finished" podID="dde1c837-949d-4349-9861-2135cda48339" containerID="0ab8b069bfdc02a918cc85cecbc6f5589451fc559f58fc6a519ad9bc1829b1d7" exitCode=0 Nov 01 23:08:40.104988 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.104897 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" event={"ID":"dde1c837-949d-4349-9861-2135cda48339","Type":"ContainerDied","Data":"0ab8b069bfdc02a918cc85cecbc6f5589451fc559f58fc6a519ad9bc1829b1d7"} Nov 01 23:08:40.569080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.569044 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:08:40.610876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.607361 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:40.610876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.607435 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:40.655886 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.625591 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\") pod \"dde1c837-949d-4349-9861-2135cda48339\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " Nov 01 23:08:40.656699 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.656669 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\") pod \"dde1c837-949d-4349-9861-2135cda48339\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " Nov 01 23:08:40.656906 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.656886 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\") pod \"dde1c837-949d-4349-9861-2135cda48339\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " Nov 01 23:08:40.657620 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.657599 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\" (UniqueName: \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\") pod \"dde1c837-949d-4349-9861-2135cda48339\" (UID: \"dde1c837-949d-4349-9861-2135cda48339\") " Nov 01 23:08:40.659762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.659737 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca" pod "dde1c837-949d-4349-9861-2135cda48339" (UID: "dde1c837-949d-4349-9861-2135cda48339"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:40.669229 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.669199 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls" pod "dde1c837-949d-4349-9861-2135cda48339" (UID: "dde1c837-949d-4349-9861-2135cda48339"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:40.669345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.669289 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf" pod "dde1c837-949d-4349-9861-2135cda48339" (UID: "dde1c837-949d-4349-9861-2135cda48339"). InnerVolumeSpecName "kube-api-access-pmdzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:40.670399 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.670376 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs" pod "dde1c837-949d-4349-9861-2135cda48339" (UID: "dde1c837-949d-4349-9861-2135cda48339"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:40.760175 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.760097 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdzf\" (UniqueName: \"kubernetes.io/projected/dde1c837-949d-4349-9861-2135cda48339-kube-api-access-pmdzf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.760371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.760353 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.760451 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.760442 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde1c837-949d-4349-9861-2135cda48339-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.760516 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:40.760505 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde1c837-949d-4349-9861-2135cda48339-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:41.113672 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.113595 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" Nov 01 23:08:41.113672 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.113613 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" event={"ID":"dde1c837-949d-4349-9861-2135cda48339","Type":"ContainerDied","Data":"5ba1159174dfd0e9ef18cc6f6d69e4be013da262119dd9934aed30400e8756e6"} Nov 01 23:08:41.113921 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.113707 5778 scope.go:122] "RemoveContainer" containerID="0ab8b069bfdc02a918cc85cecbc6f5589451fc559f58fc6a519ad9bc1829b1d7" Nov 01 23:08:41.169858 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.169823 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:08:41.174830 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.174803 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq"] Nov 01 23:08:41.599033 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.599002 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:41.599218 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:41.599054 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:42.376522 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:42.376479 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde1c837-949d-4349-9861-2135cda48339" path="/var/lib/kubelet/pods/dde1c837-949d-4349-9861-2135cda48339/volumes" Nov 01 23:08:42.598486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:42.598451 5778 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 23:08:42.598668 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:42.598506 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 23:08:43.450963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:43.450919 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c00bbf-6859867d5c-jb7jq" podUID="dde1c837-949d-4349-9861-2135cda48339" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.188:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:08:43.603818 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:43.603789 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:43.604331 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:43.604316 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:43.605091 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:43.605072 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:08:48.775958 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:08:48.775915 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:08:50.901303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:50.901276 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:08:50.959436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:50.959409 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:08:56.520887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:08:56.519906 5778 scope.go:122] "RemoveContainer" containerID="af7b77d4f6f71f75d2da71c725fbc5160a61f89f2d9193078606e2fa0323ddb2" Nov 01 23:09:11.398260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:11.397139 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:09:11.399547 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:11.399508 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerName="maas-api" containerID="cri-o://e9168a67319f1425d2510896bfc39458c7de264d71712db81b2f78955bca52e2" gracePeriod=30 Nov 01 23:09:11.583233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:11.583203 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:09:11.583656 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:11.583609 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" podUID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" containerName="payload-processing" containerID="cri-o://eda35e0f4fe49ba53872daa5de1fd9ac8936a22daed50fb8853c527381ab22a8" gracePeriod=30 Nov 01 23:09:11.907880 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:11.893569 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.175:8443/health\": dial tcp 10.132.0.175:8443: connect: connection refused" Nov 01 23:09:12.655977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.655941 5778 generic.go:356] "Generic (PLEG): container finished" podID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerID="e9168a67319f1425d2510896bfc39458c7de264d71712db81b2f78955bca52e2" exitCode=0 Nov 01 23:09:12.656781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.656749 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" event={"ID":"2d73ac8e-82bd-429b-b2b1-358d209201e9","Type":"ContainerDied","Data":"e9168a67319f1425d2510896bfc39458c7de264d71712db81b2f78955bca52e2"} Nov 01 23:09:12.665280 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.665250 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" event={"ID":"2d73ac8e-82bd-429b-b2b1-358d209201e9","Type":"ContainerDied","Data":"4f77731c87ef456b4c05d5bf14c2cf85f9e4246b88c823099bdddaa7dfb3f82e"} Nov 01 23:09:12.666297 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.666180 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f77731c87ef456b4c05d5bf14c2cf85f9e4246b88c823099bdddaa7dfb3f82e" Nov 01 23:09:12.666297 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.661172 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:09:12.667949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.667719 5778 generic.go:356] "Generic (PLEG): container finished" podID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" containerID="eda35e0f4fe49ba53872daa5de1fd9ac8936a22daed50fb8853c527381ab22a8" exitCode=0 Nov 01 23:09:12.667949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.667893 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" event={"ID":"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87","Type":"ContainerDied","Data":"eda35e0f4fe49ba53872daa5de1fd9ac8936a22daed50fb8853c527381ab22a8"} Nov 01 23:09:12.711982 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.711954 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:09:12.713302 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713273 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\") pod \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " Nov 01 23:09:12.713407 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713332 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\") pod \"2d73ac8e-82bd-429b-b2b1-358d209201e9\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " Nov 01 23:09:12.713407 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713384 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\" (UniqueName: \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\") pod \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\" (UID: \"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87\") " Nov 01 23:09:12.713494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713425 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\") pod \"2d73ac8e-82bd-429b-b2b1-358d209201e9\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " Nov 01 23:09:12.713578 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713562 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\" (UniqueName: \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\") pod \"2d73ac8e-82bd-429b-b2b1-358d209201e9\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " Nov 01 23:09:12.713625 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713606 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume" pod "e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" (UID: "e98a9dd7-b1fc-4a18-b316-78d47b0ebd87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:12.713752 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.713732 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\") pod \"2d73ac8e-82bd-429b-b2b1-358d209201e9\" (UID: \"2d73ac8e-82bd-429b-b2b1-358d209201e9\") " Nov 01 23:09:12.714197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.714094 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca" pod "2d73ac8e-82bd-429b-b2b1-358d209201e9" (UID: "2d73ac8e-82bd-429b-b2b1-358d209201e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:12.714608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.714587 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d73ac8e-82bd-429b-b2b1-358d209201e9-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.714691 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.714616 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.752874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.737354 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f" pod "e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" (UID: "e98a9dd7-b1fc-4a18-b316-78d47b0ebd87"). InnerVolumeSpecName "kube-api-access-6qj8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:12.752874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.737983 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs" pod "2d73ac8e-82bd-429b-b2b1-358d209201e9" (UID: "2d73ac8e-82bd-429b-b2b1-358d209201e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:12.752874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.748338 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls" pod "2d73ac8e-82bd-429b-b2b1-358d209201e9" (UID: "2d73ac8e-82bd-429b-b2b1-358d209201e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:12.752874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.752231 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2" pod "2d73ac8e-82bd-429b-b2b1-358d209201e9" (UID: "2d73ac8e-82bd-429b-b2b1-358d209201e9"). InnerVolumeSpecName "kube-api-access-mzkr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:12.820922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.816045 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.820922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.816081 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj8f\" (UniqueName: \"kubernetes.io/projected/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87-kube-api-access-6qj8f\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.820922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.816096 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d73ac8e-82bd-429b-b2b1-358d209201e9-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.820922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:12.816111 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzkr2\" (UniqueName: \"kubernetes.io/projected/2d73ac8e-82bd-429b-b2b1-358d209201e9-kube-api-access-mzkr2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:13.368511 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.368471 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:09:13.368744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.368689 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" podUID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" containerName="maas-api" containerID="cri-o://ede54bd91ca326dd39f7407631a58f1b6cb310cf2d73844acb39b559714d2e38" gracePeriod=30 Nov 01 23:09:13.889669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.889446 5778 generic.go:356] "Generic (PLEG): container finished" podID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" containerID="ede54bd91ca326dd39f7407631a58f1b6cb310cf2d73844acb39b559714d2e38" exitCode=0 Nov 01 23:09:13.889669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.889557 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" event={"ID":"335b477e-bf6c-42e4-b7fc-ab9628fa7094","Type":"ContainerDied","Data":"ede54bd91ca326dd39f7407631a58f1b6cb310cf2d73844acb39b559714d2e38"} Nov 01 23:09:13.926917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.920246 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8" Nov 01 23:09:13.940273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.939731 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" event={"ID":"e98a9dd7-b1fc-4a18-b316-78d47b0ebd87","Type":"ContainerDied","Data":"612131136a74cc3ef9d9ace5f33eb3884bd1819bfa7a48076333dbb3fdd7882f"} Nov 01 23:09:13.940273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.939813 5778 scope.go:122] "RemoveContainer" containerID="eda35e0f4fe49ba53872daa5de1fd9ac8936a22daed50fb8853c527381ab22a8" Nov 01 23:09:13.940273 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:13.940179 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv" Nov 01 23:09:14.190386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.190296 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:09:14.190601 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.190583 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-f66c58fb8-qrshv"] Nov 01 23:09:14.212480 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.212449 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:09:14.243274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.243243 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-337dee-56d8b5d6ff-gwvh8"] Nov 01 23:09:14.388294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.388261 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" path="/var/lib/kubelet/pods/2d73ac8e-82bd-429b-b2b1-358d209201e9/volumes" Nov 01 23:09:14.389310 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.388834 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" path="/var/lib/kubelet/pods/e98a9dd7-b1fc-4a18-b316-78d47b0ebd87/volumes" Nov 01 23:09:14.483294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.482815 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:09:14.581551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.581509 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\") pod \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " Nov 01 23:09:14.581551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.581551 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\") pod \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " Nov 01 23:09:14.581817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.581598 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\") pod \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " Nov 01 23:09:14.581817 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.581631 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\" (UniqueName: \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\") pod \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\" (UID: \"335b477e-bf6c-42e4-b7fc-ab9628fa7094\") " Nov 01 23:09:14.583002 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.582971 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca" pod "335b477e-bf6c-42e4-b7fc-ab9628fa7094" (UID: "335b477e-bf6c-42e4-b7fc-ab9628fa7094"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:14.585078 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.585054 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls" pod "335b477e-bf6c-42e4-b7fc-ab9628fa7094" (UID: "335b477e-bf6c-42e4-b7fc-ab9628fa7094"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:14.585241 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.585056 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p" pod "335b477e-bf6c-42e4-b7fc-ab9628fa7094" (UID: "335b477e-bf6c-42e4-b7fc-ab9628fa7094"). InnerVolumeSpecName "kube-api-access-g869p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:14.585343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.585294 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs" pod "335b477e-bf6c-42e4-b7fc-ab9628fa7094" (UID: "335b477e-bf6c-42e4-b7fc-ab9628fa7094"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:14.684140 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.684110 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b477e-bf6c-42e4-b7fc-ab9628fa7094-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.684352 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.684340 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.684424 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.684416 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b477e-bf6c-42e4-b7fc-ab9628fa7094-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.684482 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.684472 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g869p\" (UniqueName: \"kubernetes.io/projected/335b477e-bf6c-42e4-b7fc-ab9628fa7094-kube-api-access-g869p\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.940876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.940302 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" event={"ID":"335b477e-bf6c-42e4-b7fc-ab9628fa7094","Type":"ContainerDied","Data":"fc084426c31142df953afe3452be137d0915d0128c7b24cc1b257ee27cb05bb9"} Nov 01 23:09:14.940876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.940379 5778 scope.go:122] "RemoveContainer" containerID="ede54bd91ca326dd39f7407631a58f1b6cb310cf2d73844acb39b559714d2e38" Nov 01 23:09:14.940876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.940504 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6" Nov 01 23:09:14.983757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.983730 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:09:14.990877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:14.990445 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d2a4e-7dc66fc88d-68wm6"] Nov 01 23:09:16.291540 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.291506 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:16.292253 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292232 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:09:16.292416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292400 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4eab4d2-d13d-4d44-9331-93de4cb81141" containerName="istio-proxy" Nov 01 23:09:16.292596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292575 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerName="maas-api" Nov 01 23:09:16.292596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292594 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerName="maas-api" Nov 01 23:09:16.292777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292617 5778 cpu_manager.go:414] "containerMap: removing container" podUID="dde1c837-949d-4349-9861-2135cda48339" containerName="maas-api" Nov 01 23:09:16.292777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292623 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde1c837-949d-4349-9861-2135cda48339" containerName="maas-api" Nov 01 23:09:16.292777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292660 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" containerName="payload-processing" Nov 01 23:09:16.292777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.292668 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" containerName="payload-processing" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293185 5778 cpu_manager.go:414] "containerMap: removing container" podUID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" containerName="maas-api" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293204 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" containerName="maas-api" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293402 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d73ac8e-82bd-429b-b2b1-358d209201e9" containerName="maas-api" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293421 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde1c837-949d-4349-9861-2135cda48339" containerName="maas-api" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293430 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98a9dd7-b1fc-4a18-b316-78d47b0ebd87" containerName="payload-processing" Nov 01 23:09:16.293468 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.293437 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" containerName="maas-api" Nov 01 23:09:16.297145 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.297120 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.298797 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.298772 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:16.302726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.302709 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.309952 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.309803 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-46ce02\"" Nov 01 23:09:16.313324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.312107 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjg5\" (UniqueName: \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.313324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.312215 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.313324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.312254 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.313324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.312288 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x6w\" (UniqueName: \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.314148 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.314125 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:16.391359 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.391316 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335b477e-bf6c-42e4-b7fc-ab9628fa7094" path="/var/lib/kubelet/pods/335b477e-bf6c-42e4-b7fc-ab9628fa7094/volumes" Nov 01 23:09:16.391930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.391905 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:16.413769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.413735 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.414135 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.413815 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.414135 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.413895 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x6w\" (UniqueName: \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.414135 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.414086 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjg5\" (UniqueName: \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.421248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.415013 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.435892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.423562 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.488705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.488672 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjg5\" (UniqueName: \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.563918 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.553557 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x6w\" (UniqueName: \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\") pod \"payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:16.631885 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.631116 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:16.685792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:16.685698 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:17.062716 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:17.062675 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:17.073723 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:09:17.073322 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b256b1_7cc1_49a1_b66f_27a006b6f5bd.slice/crio-e27953693e8e12d9bd6cfd966f965ee8a1af62332d225e36bcab400ec6cf4ce6 WatchSource:0}: Error finding container e27953693e8e12d9bd6cfd966f965ee8a1af62332d225e36bcab400ec6cf4ce6: Status 404 returned error can't find the container with id e27953693e8e12d9bd6cfd966f965ee8a1af62332d225e36bcab400ec6cf4ce6 Nov 01 23:09:17.287075 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:17.287047 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:17.288881 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:09:17.288818 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313d013a_33f0_41b6_a92c_e8ec8f58b41d.slice/crio-dbaa940f1f850b55bab6f8e478f5f8791c4045a01f8509d521ce1a74105129bf WatchSource:0}: Error finding container dbaa940f1f850b55bab6f8e478f5f8791c4045a01f8509d521ce1a74105129bf: Status 404 returned error can't find the container with id dbaa940f1f850b55bab6f8e478f5f8791c4045a01f8509d521ce1a74105129bf Nov 01 23:09:17.921056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:17.920913 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 23:09:17.921624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:17.921539 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" containerID="cri-o://a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f" gracePeriod=30 Nov 01 23:09:18.130951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.124484 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" event={"ID":"313d013a-33f0-41b6-a92c-e8ec8f58b41d","Type":"ContainerStarted","Data":"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed"} Nov 01 23:09:18.130951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.124545 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" event={"ID":"313d013a-33f0-41b6-a92c-e8ec8f58b41d","Type":"ContainerStarted","Data":"dbaa940f1f850b55bab6f8e478f5f8791c4045a01f8509d521ce1a74105129bf"} Nov 01 23:09:18.130951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.125599 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:18.137605 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.137275 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" event={"ID":"71b256b1-7cc1-49a1-b66f-27a006b6f5bd","Type":"ContainerStarted","Data":"53174029b6f6830f89c89bb419841c3714cf959d3258556014ed1ce7f9ae64ed"} Nov 01 23:09:18.138666 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.138640 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" event={"ID":"71b256b1-7cc1-49a1-b66f-27a006b6f5bd","Type":"ContainerStarted","Data":"e27953693e8e12d9bd6cfd966f965ee8a1af62332d225e36bcab400ec6cf4ce6"} Nov 01 23:09:18.138970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.138945 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:18.198274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.193455 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" podStartSLOduration=2.193435368 podStartE2EDuration="2.193435368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:18.183696756 +0000 UTC m=+2727.959683587" watchObservedRunningTime="2026-11-01 23:09:18.193435368 +0000 UTC m=+2727.969422200" Nov 01 23:09:18.198274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:18.194941 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" podStartSLOduration=2.194929438 podStartE2EDuration="2.194929438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:18.168082621 +0000 UTC m=+2727.944069450" watchObservedRunningTime="2026-11-01 23:09:18.194929438 +0000 UTC m=+2727.970916268" Nov 01 23:09:18.774450 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:18.774408 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:09:19.439358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:19.439314 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:09:19.439837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:19.439385 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:09:20.337747 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.337715 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:20.342745 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.342724 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.360895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.360863 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441212 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441317 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441379 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441409 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441445 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441529 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76src\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441583 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441621 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.441881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.441674 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.543719 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.543677 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.543990 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.543778 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.543990 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.543913 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.544160 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.543945 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.544371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.544341 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.545087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.544768 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.545087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545048 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.545322 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545302 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.545500 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545486 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76src\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.546168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545792 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.546168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545841 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.546168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.545927 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.547188 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.547148 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.547451 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.547425 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.547713 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.547689 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.548265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.548240 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.557986 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.555501 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.557986 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.556521 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76src\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.663958 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.663924 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:20.877781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.877745 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:20.884161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.883670 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:09:20.884161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.883920 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" containerID="cri-o://9781e802180009aa35d5e94fea685c1e4d5c0b35a6b01e8987842a10f7365f5d" gracePeriod=30 Nov 01 23:09:20.993436 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:20.993361 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:20.997674 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:09:20.997640 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f0528a_972f_484a_b74d_1112107b4e0a.slice/crio-ceed182a23613f1a35135fc2a730fd003de7f49de1786ec8f577bc413e45dea6 WatchSource:0}: Error finding container ceed182a23613f1a35135fc2a730fd003de7f49de1786ec8f577bc413e45dea6: Status 404 returned error can't find the container with id ceed182a23613f1a35135fc2a730fd003de7f49de1786ec8f577bc413e45dea6 Nov 01 23:09:21.002374 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:21.002333 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:21.002472 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:21.002425 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:21.002525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:21.002470 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:21.233624 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:21.233544 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" event={"ID":"a8f0528a-972f-484a-b74d-1112107b4e0a","Type":"ContainerStarted","Data":"ceed182a23613f1a35135fc2a730fd003de7f49de1786ec8f577bc413e45dea6"} Nov 01 23:09:22.263438 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:22.246991 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" podUID="a8f0528a-972f-484a-b74d-1112107b4e0a" containerName="istio-proxy" containerID="cri-o://18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337" gracePeriod=30 Nov 01 23:09:22.263438 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:22.248886 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" event={"ID":"a8f0528a-972f-484a-b74d-1112107b4e0a","Type":"ContainerStarted","Data":"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337"} Nov 01 23:09:22.299819 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:22.278409 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" podStartSLOduration=2.278392144 podStartE2EDuration="2.278392144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:22.270451967 +0000 UTC m=+2732.046438977" watchObservedRunningTime="2026-11-01 23:09:22.278392144 +0000 UTC m=+2732.054378972" Nov 01 23:09:23.222573 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.222544 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 23:09:23.257615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.257580 5778 generic.go:356] "Generic (PLEG): container finished" podID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerID="5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9" exitCode=0 Nov 01 23:09:23.257804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.257661 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerDied","Data":"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9"} Nov 01 23:09:23.258151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.258131 5778 scope.go:122] "RemoveContainer" containerID="5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9" Nov 01 23:09:23.260396 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.260362 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:23.260542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.260451 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:23.260542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.260505 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:09:23.265430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.264946 5778 generic.go:356] "Generic (PLEG): container finished" podID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerID="a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f" exitCode=0 Nov 01 23:09:23.265430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.265112 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerDied","Data":"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f"} Nov 01 23:09:23.265430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.265156 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" event={"ID":"ea46be7d-a060-4d72-91db-e4aabfbd2bec","Type":"ContainerDied","Data":"d02804ebbb69b5576383dfff54d065e1c9389225c7d61825dba00c52a9a1888c"} Nov 01 23:09:23.265430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.265236 5778 scope.go:122] "RemoveContainer" containerID="a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f" Nov 01 23:09:23.265430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.265392 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp" Nov 01 23:09:23.280930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.280908 5778 scope.go:122] "RemoveContainer" containerID="640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27" Nov 01 23:09:23.307738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.307710 5778 scope.go:122] "RemoveContainer" containerID="a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f" Nov 01 23:09:23.308499 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:23.308319 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f\": container with ID starting with a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f not found: ID does not exist" containerID="a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f" Nov 01 23:09:23.308592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.308513 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f"} err="failed to get container status \"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f\": rpc error: code = NotFound desc = could not find container \"a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f\": container with ID starting with a1074685bd0f58bbad0df6c83ff96ab843391b4ceeb9f786ac85a2854df6025f not found: ID does not exist" Nov 01 23:09:23.308592 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.308537 5778 scope.go:122] "RemoveContainer" containerID="640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27" Nov 01 23:09:23.309166 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:23.308842 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27\": container with ID starting with 640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27 not found: ID does not exist" containerID="640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27" Nov 01 23:09:23.309286 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.309170 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27"} err="failed to get container status \"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27\": rpc error: code = NotFound desc = could not find container \"640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27\": container with ID starting with 640fad65bb22b510914c671f5bcd728784ddf644edd7c76822b7792dbd1e2b27 not found: ID does not exist" Nov 01 23:09:23.318917 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.318894 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346589 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346712 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346761 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346809 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346875 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.346993 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.347051 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.347102 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.347200 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\") pod \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\" (UID: \"ea46be7d-a060-4d72-91db-e4aabfbd2bec\") " Nov 01 23:09:23.350274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.348699 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.352979 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.352166 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.352979 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.352257 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.352979 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.352334 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.352979 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.352612 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:23.357119 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.357094 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.361146 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.361104 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:23.361634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.361587 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:23.363649 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.363343 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl" pod "ea46be7d-a060-4d72-91db-e4aabfbd2bec" (UID: "ea46be7d-a060-4d72-91db-e4aabfbd2bec"). InnerVolumeSpecName "kube-api-access-5kngl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460179 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460212 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460228 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460244 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kngl\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-kube-api-access-5kngl\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460262 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460277 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460290 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460304 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.460690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.460317 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea46be7d-a060-4d72-91db-e4aabfbd2bec-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.593584 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.593541 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 23:09:23.597163 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:23.597124 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-dpzfp"] Nov 01 23:09:24.275993 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:24.275883 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerStarted","Data":"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e"} Nov 01 23:09:24.406564 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:24.406532 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" path="/var/lib/kubelet/pods/ea46be7d-a060-4d72-91db-e4aabfbd2bec/volumes" Nov 01 23:09:24.864535 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:24.864492 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:09:24.865783 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:24.865753 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 23:09:24.865951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:24.865814 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 23:09:25.864172 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:25.864136 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 23:09:25.864684 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:25.864193 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 23:09:26.300653 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.300616 5778 generic.go:356] "Generic (PLEG): container finished" podID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerID="9781e802180009aa35d5e94fea685c1e4d5c0b35a6b01e8987842a10f7365f5d" exitCode=0 Nov 01 23:09:26.301126 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.301098 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" event={"ID":"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7","Type":"ContainerDied","Data":"9781e802180009aa35d5e94fea685c1e4d5c0b35a6b01e8987842a10f7365f5d"} Nov 01 23:09:26.301244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.301146 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" event={"ID":"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7","Type":"ContainerDied","Data":"eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44"} Nov 01 23:09:26.301244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.301163 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb8da307fb6e9f00b1250b0171f653d8b60a4fa3b19bbbd5791765174b8fb44" Nov 01 23:09:26.311122 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.311090 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364193 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364239 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364305 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364338 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364428 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364446 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364476 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364549 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.364876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.364577 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\") pod \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\" (UID: \"fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7\") " Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.366065 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.366326 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.366422 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.371613 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.372206 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.375349 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.372453 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.386147 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.386107 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.386387 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.386336 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:26.388282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.388258 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l" pod "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" (UID: "fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7"). InnerVolumeSpecName "kube-api-access-vng8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467120 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467152 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467165 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467179 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467191 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467208 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467204 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vng8l\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-kube-api-access-vng8l\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467666 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467219 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467666 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467232 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.467666 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.467243 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.864285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.864204 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 23:09:26.864285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:26.864265 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 23:09:27.308485 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.308453 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m" Nov 01 23:09:27.375873 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.373594 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:09:27.379440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.379410 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-4t84m"] Nov 01 23:09:27.591730 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.591699 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:27.693285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693246 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693490 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693327 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693559 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693495 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693559 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693548 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693677 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693599 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693737 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693687 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693737 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693731 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693797 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.693928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.693878 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\") pod \"a8f0528a-972f-484a-b74d-1112107b4e0a\" (UID: \"a8f0528a-972f-484a-b74d-1112107b4e0a\") " Nov 01 23:09:27.697795 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.697762 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:27.698021 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.697997 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:27.698093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.698063 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:27.698152 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.698109 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:27.698527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.698503 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:27.699991 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.699963 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:27.701928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.701843 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:27.702197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.702177 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:27.702443 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.702422 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src" pod "a8f0528a-972f-484a-b74d-1112107b4e0a" (UID: "a8f0528a-972f-484a-b74d-1112107b4e0a"). InnerVolumeSpecName "kube-api-access-76src". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:27.797908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797874 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.797908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797910 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797927 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797940 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797957 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76src\" (UniqueName: \"kubernetes.io/projected/a8f0528a-972f-484a-b74d-1112107b4e0a-kube-api-access-76src\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797971 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.797984 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f0528a-972f-484a-b74d-1112107b4e0a-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.798000 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f0528a-972f-484a-b74d-1112107b4e0a-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.798117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.798013 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f0528a-972f-484a-b74d-1112107b4e0a-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.865548 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.865448 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 23:09:27.865548 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:27.865531 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 23:09:28.347516 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.347479 5778 generic.go:356] "Generic (PLEG): container finished" podID="a8f0528a-972f-484a-b74d-1112107b4e0a" containerID="18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337" exitCode=0 Nov 01 23:09:28.348080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.347921 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" Nov 01 23:09:28.348244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.347946 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" event={"ID":"a8f0528a-972f-484a-b74d-1112107b4e0a","Type":"ContainerDied","Data":"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337"} Nov 01 23:09:28.349219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.349189 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2" event={"ID":"a8f0528a-972f-484a-b74d-1112107b4e0a","Type":"ContainerDied","Data":"ceed182a23613f1a35135fc2a730fd003de7f49de1786ec8f577bc413e45dea6"} Nov 01 23:09:28.349345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.349240 5778 scope.go:122] "RemoveContainer" containerID="18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337" Nov 01 23:09:28.363304 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.363281 5778 scope.go:122] "RemoveContainer" containerID="18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337" Nov 01 23:09:28.363567 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:28.363544 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337\": container with ID starting with 18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337 not found: ID does not exist" containerID="18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337" Nov 01 23:09:28.363649 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.363575 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337"} err="failed to get container status \"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337\": rpc error: code = NotFound desc = could not find container \"18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337\": container with ID starting with 18c4abddd8329ef5d6e766963e1a74b586e67f1a6cc8d82bde7dac0875763337 not found: ID does not exist" Nov 01 23:09:28.377241 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.377183 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" path="/var/lib/kubelet/pods/fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7/volumes" Nov 01 23:09:28.383210 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.383182 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:28.383324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.383225 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-nmqh2"] Nov 01 23:09:28.749262 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.749224 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 23:09:28.762615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.762542 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 23:09:28.864623 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.863989 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 23:09:28.864623 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:28.864043 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 23:09:29.931057 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:29.931023 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:09:29.931703 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:29.931671 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:09:29.932340 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:29.932319 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:09:30.159016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.158980 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:30.159357 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.159334 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:30.377129 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.377051 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f0528a-972f-484a-b74d-1112107b4e0a" path="/var/lib/kubelet/pods/a8f0528a-972f-484a-b74d-1112107b4e0a/volumes" Nov 01 23:09:30.831936 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.831900 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:09:30.832723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832691 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.832723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832719 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832732 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a8f0528a-972f-484a-b74d-1112107b4e0a" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832738 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f0528a-972f-484a-b74d-1112107b4e0a" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832754 5778 cpu_manager.go:414] "containerMap: removing container" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832760 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832777 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832782 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832940 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832951 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f0528a-972f-484a-b74d-1112107b4e0a" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832959 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec5eae9-9d6c-4a46-82f7-70abf5eb9ec7" containerName="istio-proxy" Nov 01 23:09:30.833000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.832967 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea46be7d-a060-4d72-91db-e4aabfbd2bec" containerName="istio-proxy" Nov 01 23:09:30.837927 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.837905 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:30.842579 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.842542 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-46ce02-serving-cert\"" Nov 01 23:09:30.846133 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:30.846114 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:09:31.007115 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.007074 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.007529 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.007130 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm65w\" (UniqueName: \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.007529 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.007296 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.007529 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.007351 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.109343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.109249 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.109343 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.109307 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.109593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.109432 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.109593 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.109456 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm65w\" (UniqueName: \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.110083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.110054 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.112197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.112173 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.112301 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.112250 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.128045 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.128013 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm65w\" (UniqueName: \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\") pod \"maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.154250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.154229 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:31.568689 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.568652 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:09:31.596635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:31.592356 5778 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:09:32.426128 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:32.426035 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" event={"ID":"d448a01a-e368-4968-8912-9d8b1cb19f8f","Type":"ContainerStarted","Data":"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11"} Nov 01 23:09:32.426697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:32.426133 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" event={"ID":"d448a01a-e368-4968-8912-9d8b1cb19f8f","Type":"ContainerStarted","Data":"d9d26441c36e737d0b1fee18b340285bda6bf4b7e3aa9c97dcd8756aee31a616"} Nov 01 23:09:32.426697 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:32.426339 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:32.445425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:32.445370 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" podStartSLOduration=1.877501176 podStartE2EDuration="2.445353835s" totalImagesPullingTime="567.852659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:32.444175626 +0000 UTC m=+2742.220162458" watchObservedRunningTime="2026-11-01 23:09:32.445353835 +0000 UTC m=+2742.221340663" Nov 01 23:09:33.770878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:33.762921 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" containerID="cri-o://a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962" gracePeriod=25 Nov 01 23:09:33.770878 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:33.763088 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" containerID="cri-o://f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50" gracePeriod=25 Nov 01 23:09:35.853095 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:35.853052 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:36.858826 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:36.858786 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:38.438543 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:38.438516 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:09:45.854017 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:45.853972 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:46.859353 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:46.859311 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:48.775158 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:48.775124 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:09:50.019788 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:50.019754 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:50.037760 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:50.037724 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:54.010170 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.010108 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 23:09:54.160877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.156517 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\" (UniqueName: \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\") pod \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " Nov 01 23:09:54.160877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.156571 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\") pod \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\" (UID: \"58a753a9-2199-49ce-a1e2-3292ec66f1ab\") " Nov 01 23:09:54.205877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.196388 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume" pod "58a753a9-2199-49ce-a1e2-3292ec66f1ab" (UID: "58a753a9-2199-49ce-a1e2-3292ec66f1ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:54.205877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.204788 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a753a9-2199-49ce-a1e2-3292ec66f1ab-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:54.211921 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.209023 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84" pod "58a753a9-2199-49ce-a1e2-3292ec66f1ab" (UID: "58a753a9-2199-49ce-a1e2-3292ec66f1ab"). InnerVolumeSpecName "kube-api-access-qkv84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:54.273000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.267074 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 23:09:54.306304 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.306266 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\") pod \"a429d996-5732-4237-878a-cb5c498a7570\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " Nov 01 23:09:54.306504 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.306368 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\" (UniqueName: \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\") pod \"a429d996-5732-4237-878a-cb5c498a7570\" (UID: \"a429d996-5732-4237-878a-cb5c498a7570\") " Nov 01 23:09:54.306702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.306675 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume" pod "a429d996-5732-4237-878a-cb5c498a7570" (UID: "a429d996-5732-4237-878a-cb5c498a7570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:54.306923 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.306901 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkv84\" (UniqueName: \"kubernetes.io/projected/58a753a9-2199-49ce-a1e2-3292ec66f1ab-kube-api-access-qkv84\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:54.306998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.306929 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429d996-5732-4237-878a-cb5c498a7570-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:54.309901 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.309103 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l" pod "a429d996-5732-4237-878a-cb5c498a7570" (UID: "a429d996-5732-4237-878a-cb5c498a7570"). InnerVolumeSpecName "kube-api-access-bkf6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:54.408249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.408219 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkf6l\" (UniqueName: \"kubernetes.io/projected/a429d996-5732-4237-878a-cb5c498a7570-kube-api-access-bkf6l\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:54.678315 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.678284 5778 generic.go:356] "Generic (PLEG): container finished" podID="a429d996-5732-4237-878a-cb5c498a7570" containerID="f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50" exitCode=0 Nov 01 23:09:54.678527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.678328 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" Nov 01 23:09:54.678527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.678363 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" event={"ID":"a429d996-5732-4237-878a-cb5c498a7570","Type":"ContainerDied","Data":"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50"} Nov 01 23:09:54.678527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.678426 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp" event={"ID":"a429d996-5732-4237-878a-cb5c498a7570","Type":"ContainerDied","Data":"c50244f509021f92dbb28716160427d08ce0e2e57dec393f5d400c78531baee4"} Nov 01 23:09:54.678527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.678458 5778 scope.go:122] "RemoveContainer" containerID="f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50" Nov 01 23:09:54.680167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.680142 5778 generic.go:356] "Generic (PLEG): container finished" podID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerID="a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962" exitCode=0 Nov 01 23:09:54.680287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.680179 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" Nov 01 23:09:54.680287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.680175 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" event={"ID":"58a753a9-2199-49ce-a1e2-3292ec66f1ab","Type":"ContainerDied","Data":"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962"} Nov 01 23:09:54.680405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.680303 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp" event={"ID":"58a753a9-2199-49ce-a1e2-3292ec66f1ab","Type":"ContainerDied","Data":"9e200e6d1f1b4c5db3f2c1b6fc52cec2185fcd43ee13bbb76e730ac476d18cc6"} Nov 01 23:09:54.690112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.690093 5778 scope.go:122] "RemoveContainer" containerID="f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50" Nov 01 23:09:54.690369 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:54.690348 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50\": container with ID starting with f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50 not found: ID does not exist" containerID="f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50" Nov 01 23:09:54.690416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.690377 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50"} err="failed to get container status \"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50\": rpc error: code = NotFound desc = could not find container \"f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50\": container with ID starting with f29e0365542cf8f76626d62329a7ae212f2eebcf844e23e7927c5bdb6b716d50 not found: ID does not exist" Nov 01 23:09:54.690416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.690391 5778 scope.go:122] "RemoveContainer" containerID="a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962" Nov 01 23:09:54.701086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.701067 5778 scope.go:122] "RemoveContainer" containerID="a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962" Nov 01 23:09:54.701408 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:54.701385 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962\": container with ID starting with a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962 not found: ID does not exist" containerID="a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962" Nov 01 23:09:54.701503 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.701416 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962"} err="failed to get container status \"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962\": rpc error: code = NotFound desc = could not find container \"a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962\": container with ID starting with a9f008e6572a5f980eba1b5aabf618185e6cc5a76124e727e1105e152dfe2962 not found: ID does not exist" Nov 01 23:09:54.702622 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.702602 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 23:09:54.708711 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.708689 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-927b68-595fc869f-cj6zp"] Nov 01 23:09:54.722001 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.721975 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 23:09:54.727278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:54.727255 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-927b68-6dbfc59b95-b5trp"] Nov 01 23:09:55.020487 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.020396 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" podUID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" containerName="payload-processing" containerID="cri-o://598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed" gracePeriod=25 Nov 01 23:09:55.039380 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.039342 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" podUID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" containerName="payload-pre-processing" containerID="cri-o://53174029b6f6830f89c89bb419841c3714cf959d3258556014ed1ce7f9ae64ed" gracePeriod=25 Nov 01 23:09:55.246932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.245841 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 23:09:55.246932 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.246119 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" podUID="8fe315b0-6760-49d9-b333-5531f0c66cad" containerName="maas-api" containerID="cri-o://05f46f491c5e9304447938fcd3039ec96757940d1e4cd51356ea19673cba99bf" gracePeriod=30 Nov 01 23:09:55.561933 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.555060 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:55.697060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.695906 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\" (UniqueName: \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\") pod \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " Nov 01 23:09:55.697060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.696224 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\") pod \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\" (UID: \"313d013a-33f0-41b6-a92c-e8ec8f58b41d\") " Nov 01 23:09:55.697792 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.697464 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume" pod "313d013a-33f0-41b6-a92c-e8ec8f58b41d" (UID: "313d013a-33f0-41b6-a92c-e8ec8f58b41d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:55.708908 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.708839 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w" pod "313d013a-33f0-41b6-a92c-e8ec8f58b41d" (UID: "313d013a-33f0-41b6-a92c-e8ec8f58b41d"). InnerVolumeSpecName "kube-api-access-g5x6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:55.855724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.855676 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313d013a-33f0-41b6-a92c-e8ec8f58b41d-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:55.855724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.855728 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5x6w\" (UniqueName: \"kubernetes.io/projected/313d013a-33f0-41b6-a92c-e8ec8f58b41d-kube-api-access-g5x6w\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:55.882659 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.882611 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:55.891520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.891487 5778 generic.go:356] "Generic (PLEG): container finished" podID="8fe315b0-6760-49d9-b333-5531f0c66cad" containerID="05f46f491c5e9304447938fcd3039ec96757940d1e4cd51356ea19673cba99bf" exitCode=0 Nov 01 23:09:55.891974 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.891949 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" event={"ID":"8fe315b0-6760-49d9-b333-5531f0c66cad","Type":"ContainerDied","Data":"05f46f491c5e9304447938fcd3039ec96757940d1e4cd51356ea19673cba99bf"} Nov 01 23:09:55.894268 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.894245 5778 generic.go:356] "Generic (PLEG): container finished" podID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" containerID="598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed" exitCode=0 Nov 01 23:09:55.894664 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.894640 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" Nov 01 23:09:55.894916 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.894543 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" event={"ID":"313d013a-33f0-41b6-a92c-e8ec8f58b41d","Type":"ContainerDied","Data":"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed"} Nov 01 23:09:55.895589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.895564 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp" event={"ID":"313d013a-33f0-41b6-a92c-e8ec8f58b41d","Type":"ContainerDied","Data":"dbaa940f1f850b55bab6f8e478f5f8791c4045a01f8509d521ce1a74105129bf"} Nov 01 23:09:55.895779 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.895712 5778 scope.go:122] "RemoveContainer" containerID="598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed" Nov 01 23:09:55.902187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.902164 5778 generic.go:356] "Generic (PLEG): container finished" podID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" containerID="53174029b6f6830f89c89bb419841c3714cf959d3258556014ed1ce7f9ae64ed" exitCode=0 Nov 01 23:09:55.902440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.902416 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" event={"ID":"71b256b1-7cc1-49a1-b66f-27a006b6f5bd","Type":"ContainerDied","Data":"53174029b6f6830f89c89bb419841c3714cf959d3258556014ed1ce7f9ae64ed"} Nov 01 23:09:55.902673 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.902657 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68" Nov 01 23:09:55.921034 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.921007 5778 scope.go:122] "RemoveContainer" containerID="598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed" Nov 01 23:09:55.921403 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:09:55.921362 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed\": container with ID starting with 598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed not found: ID does not exist" containerID="598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed" Nov 01 23:09:55.921508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.921401 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed"} err="failed to get container status \"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed\": rpc error: code = NotFound desc = could not find container \"598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed\": container with ID starting with 598aee18f0f412c54dfbb01df97d8949a9c5c5dec1775bdb34db5735b80078ed not found: ID does not exist" Nov 01 23:09:55.921508 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.921423 5778 scope.go:122] "RemoveContainer" containerID="53174029b6f6830f89c89bb419841c3714cf959d3258556014ed1ce7f9ae64ed" Nov 01 23:09:55.957755 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.957716 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\") pod \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " Nov 01 23:09:55.958213 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.958175 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\" (UniqueName: \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\") pod \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\" (UID: \"71b256b1-7cc1-49a1-b66f-27a006b6f5bd\") " Nov 01 23:09:55.959259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.958412 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume" pod "71b256b1-7cc1-49a1-b66f-27a006b6f5bd" (UID: "71b256b1-7cc1-49a1-b66f-27a006b6f5bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:55.959395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.958451 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:55.959519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.959493 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:55.961926 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.961780 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-66d6b68d67-8zxhp"] Nov 01 23:09:55.964748 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.964721 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5" pod "71b256b1-7cc1-49a1-b66f-27a006b6f5bd" (UID: "71b256b1-7cc1-49a1-b66f-27a006b6f5bd"). InnerVolumeSpecName "kube-api-access-tkjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:55.999954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:55.999926 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 23:09:56.060569 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.060537 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\") pod \"8fe315b0-6760-49d9-b333-5531f0c66cad\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " Nov 01 23:09:56.061072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.060662 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\") pod \"8fe315b0-6760-49d9-b333-5531f0c66cad\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " Nov 01 23:09:56.061072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.060841 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\" (UniqueName: \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\") pod \"8fe315b0-6760-49d9-b333-5531f0c66cad\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " Nov 01 23:09:56.061072 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.060988 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\") pod \"8fe315b0-6760-49d9-b333-5531f0c66cad\" (UID: \"8fe315b0-6760-49d9-b333-5531f0c66cad\") " Nov 01 23:09:56.061506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.061485 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjg5\" (UniqueName: \"kubernetes.io/projected/71b256b1-7cc1-49a1-b66f-27a006b6f5bd-kube-api-access-tkjg5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.063553 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.063521 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs" pod "8fe315b0-6760-49d9-b333-5531f0c66cad" (UID: "8fe315b0-6760-49d9-b333-5531f0c66cad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:56.063930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.063837 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls" pod "8fe315b0-6760-49d9-b333-5531f0c66cad" (UID: "8fe315b0-6760-49d9-b333-5531f0c66cad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:56.064018 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.063924 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca" pod "8fe315b0-6760-49d9-b333-5531f0c66cad" (UID: "8fe315b0-6760-49d9-b333-5531f0c66cad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:56.064720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.064695 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf" pod "8fe315b0-6760-49d9-b333-5531f0c66cad" (UID: "8fe315b0-6760-49d9-b333-5531f0c66cad"). InnerVolumeSpecName "kube-api-access-79ntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:56.162820 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.162782 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ntf\" (UniqueName: \"kubernetes.io/projected/8fe315b0-6760-49d9-b333-5531f0c66cad-kube-api-access-79ntf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.162820 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.162816 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.162820 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.162827 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe315b0-6760-49d9-b333-5531f0c66cad-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.163125 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.162836 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe315b0-6760-49d9-b333-5531f0c66cad-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.230398 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.230365 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:56.235422 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.235395 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-675c8c954c-ftx68"] Nov 01 23:09:56.379252 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.379162 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" path="/var/lib/kubelet/pods/313d013a-33f0-41b6-a92c-e8ec8f58b41d/volumes" Nov 01 23:09:56.379722 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.379699 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" path="/var/lib/kubelet/pods/58a753a9-2199-49ce-a1e2-3292ec66f1ab/volumes" Nov 01 23:09:56.380242 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.380220 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" path="/var/lib/kubelet/pods/71b256b1-7cc1-49a1-b66f-27a006b6f5bd/volumes" Nov 01 23:09:56.380684 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.380664 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a429d996-5732-4237-878a-cb5c498a7570" path="/var/lib/kubelet/pods/a429d996-5732-4237-878a-cb5c498a7570/volumes" Nov 01 23:09:56.618227 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.618198 5778 scope.go:122] "RemoveContainer" containerID="e9168a67319f1425d2510896bfc39458c7de264d71712db81b2f78955bca52e2" Nov 01 23:09:56.631629 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.631602 5778 scope.go:122] "RemoveContainer" containerID="05f46f491c5e9304447938fcd3039ec96757940d1e4cd51356ea19673cba99bf" Nov 01 23:09:56.911955 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.911914 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" event={"ID":"8fe315b0-6760-49d9-b333-5531f0c66cad","Type":"ContainerDied","Data":"8ca10637696bf8d4fb03f9fa9415a8c8478af9df76027ee14b92dc9f6350bc6f"} Nov 01 23:09:56.912164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.912098 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh" Nov 01 23:09:56.950809 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.950771 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 23:09:56.955542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:56.955515 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-927b68-5fcf68b47-699zh"] Nov 01 23:09:58.375260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:09:58.375220 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe315b0-6760-49d9-b333-5531f0c66cad" path="/var/lib/kubelet/pods/8fe315b0-6760-49d9-b333-5531f0c66cad/volumes" Nov 01 23:10:03.549872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:03.545103 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:10:03.549872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:03.545349 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" containerID="cri-o://6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d" gracePeriod=30 Nov 01 23:10:07.221579 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:07.221534 5778 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:10:07.222095 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:07.221614 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:10:09.078871 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.075093 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:10:09.079911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.079481 5778 generic.go:356] "Generic (PLEG): container finished" podID="6688b26c-1109-45d1-b84b-71d69646e29d" containerID="6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d" exitCode=0 Nov 01 23:10:09.079911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.079523 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" Nov 01 23:10:09.079911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.079573 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" event={"ID":"6688b26c-1109-45d1-b84b-71d69646e29d","Type":"ContainerDied","Data":"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d"} Nov 01 23:10:09.079911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.079643 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp" event={"ID":"6688b26c-1109-45d1-b84b-71d69646e29d","Type":"ContainerDied","Data":"5107e610ba1c68d850aa20d29d84dc2e50b2ceab4c4876ebdb31845ce18da0e3"} Nov 01 23:10:09.079911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.079675 5778 scope.go:122] "RemoveContainer" containerID="6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d" Nov 01 23:10:09.090787 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.090766 5778 scope.go:122] "RemoveContainer" containerID="6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d" Nov 01 23:10:09.091099 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:09.091078 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d\": container with ID starting with 6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d not found: ID does not exist" containerID="6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d" Nov 01 23:10:09.091194 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.091108 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d"} err="failed to get container status \"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d\": rpc error: code = NotFound desc = could not find container \"6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d\": container with ID starting with 6343d2263fc244e6877bad290819743e45631e1c3be0808bfb8201712370eb7d not found: ID does not exist" Nov 01 23:10:09.256293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256262 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256442 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256496 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256648 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256532 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256648 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256575 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256648 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256621 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256692 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256725 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.256807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256793 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\") pod \"6688b26c-1109-45d1-b84b-71d69646e29d\" (UID: \"6688b26c-1109-45d1-b84b-71d69646e29d\") " Nov 01 23:10:09.257008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256934 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.257008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.256948 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.257123 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257026 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:09.257358 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257332 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.257447 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257371 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.257608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257579 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.257608 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257610 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6688b26c-1109-45d1-b84b-71d69646e29d-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.257879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257625 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.257879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257640 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.257879 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.257655 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.258937 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.258903 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.259160 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.259125 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.259660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.259641 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "kube-api-access-fm8rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.259802 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.259786 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo" pod "6688b26c-1109-45d1-b84b-71d69646e29d" (UID: "6688b26c-1109-45d1-b84b-71d69646e29d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:09.359287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.359198 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6688b26c-1109-45d1-b84b-71d69646e29d-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.359287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.359231 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8rp\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-kube-api-access-fm8rp\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.359287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.359242 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6688b26c-1109-45d1-b84b-71d69646e29d-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.359287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.359251 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6688b26c-1109-45d1-b84b-71d69646e29d-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.409142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.409107 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:10:09.409312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:09.409158 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-6cbrp"] Nov 01 23:10:10.374814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:10.374768 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" path="/var/lib/kubelet/pods/6688b26c-1109-45d1-b84b-71d69646e29d/volumes" Nov 01 23:10:14.233290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:14.233254 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 23:10:18.781363 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:18.781327 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:10:29.216115 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.216012 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:10:29.216628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.216271 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerName="maas-api" containerID="cri-o://c191edc5be38a67a80475f378304cd1dcce55bcd20652eb7e373d0a6f6423fab" gracePeriod=30 Nov 01 23:10:29.258116 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.258081 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" containerID="cri-o://b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73" gracePeriod=45 Nov 01 23:10:29.405119 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.405063 5778 generic.go:356] "Generic (PLEG): container finished" podID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerID="c191edc5be38a67a80475f378304cd1dcce55bcd20652eb7e373d0a6f6423fab" exitCode=0 Nov 01 23:10:29.405585 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.405517 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" event={"ID":"9cce4af7-41f1-46ef-a412-08a0203f4c5f","Type":"ContainerDied","Data":"c191edc5be38a67a80475f378304cd1dcce55bcd20652eb7e373d0a6f6423fab"} Nov 01 23:10:29.876512 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.876482 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:10:29.942427 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.942398 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 23:10:29.968762 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.968729 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\") pod \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " Nov 01 23:10:29.969047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.969027 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\" (UniqueName: \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\") pod \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " Nov 01 23:10:29.969305 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.969277 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\") pod \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " Nov 01 23:10:29.969420 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.969323 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\") pod \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\" (UID: \"9cce4af7-41f1-46ef-a412-08a0203f4c5f\") " Nov 01 23:10:29.970234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.970209 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca" pod "9cce4af7-41f1-46ef-a412-08a0203f4c5f" (UID: "9cce4af7-41f1-46ef-a412-08a0203f4c5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:29.972628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.972603 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj" pod "9cce4af7-41f1-46ef-a412-08a0203f4c5f" (UID: "9cce4af7-41f1-46ef-a412-08a0203f4c5f"). InnerVolumeSpecName "kube-api-access-vlmdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.972768 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.972628 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs" pod "9cce4af7-41f1-46ef-a412-08a0203f4c5f" (UID: "9cce4af7-41f1-46ef-a412-08a0203f4c5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:29.973300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:29.973267 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls" pod "9cce4af7-41f1-46ef-a412-08a0203f4c5f" (UID: "9cce4af7-41f1-46ef-a412-08a0203f4c5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:30.071049 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071019 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071063 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071128 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071157 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071221 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071326 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:30.071470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071353 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\" (UniqueName: \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\") pod \"c478da01-6384-454f-b3ce-e521652a51ff\" (UID: \"c478da01-6384-454f-b3ce-e521652a51ff\") " Nov 01 23:10:30.071589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071515 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:30.071636 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.071611 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:30.072178 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072150 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072181 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072196 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072210 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072224 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmdj\" (UniqueName: \"kubernetes.io/projected/9cce4af7-41f1-46ef-a412-08a0203f4c5f-kube-api-access-vlmdj\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072240 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cce4af7-41f1-46ef-a412-08a0203f4c5f-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.072325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.072256 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cce4af7-41f1-46ef-a412-08a0203f4c5f-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.073596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.073559 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:30.073711 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.073639 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:30.073711 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.073691 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz" pod "c478da01-6384-454f-b3ce-e521652a51ff" (UID: "c478da01-6384-454f-b3ce-e521652a51ff"). InnerVolumeSpecName "kube-api-access-nrqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:30.173744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.173714 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqxz\" (UniqueName: \"kubernetes.io/projected/c478da01-6384-454f-b3ce-e521652a51ff-kube-api-access-nrqxz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.173744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.173743 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c478da01-6384-454f-b3ce-e521652a51ff-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.174011 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.173752 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c478da01-6384-454f-b3ce-e521652a51ff-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.413494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.413472 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" Nov 01 23:10:30.413930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.413509 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" event={"ID":"9cce4af7-41f1-46ef-a412-08a0203f4c5f","Type":"ContainerDied","Data":"20bdf47375b0edeb20d780edd207a56a1b54018d4e25e350203597ff940e26ae"} Nov 01 23:10:30.413930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.413572 5778 scope.go:122] "RemoveContainer" containerID="c191edc5be38a67a80475f378304cd1dcce55bcd20652eb7e373d0a6f6423fab" Nov 01 23:10:30.416313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.416104 5778 generic.go:356] "Generic (PLEG): container finished" podID="c478da01-6384-454f-b3ce-e521652a51ff" containerID="b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73" exitCode=0 Nov 01 23:10:30.416313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.416124 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" Nov 01 23:10:30.416313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.416174 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" event={"ID":"c478da01-6384-454f-b3ce-e521652a51ff","Type":"ContainerDied","Data":"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73"} Nov 01 23:10:30.416313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.416206 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" event={"ID":"c478da01-6384-454f-b3ce-e521652a51ff","Type":"ContainerDied","Data":"f544e27583e8a8905d7708df3c6f01008460a3bb85469b168898576fc7e6bad5"} Nov 01 23:10:30.429615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.429587 5778 scope.go:122] "RemoveContainer" containerID="b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73" Nov 01 23:10:30.435476 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.435453 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 23:10:30.440061 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.440029 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9"] Nov 01 23:10:30.441996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.441973 5778 scope.go:122] "RemoveContainer" containerID="b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73" Nov 01 23:10:30.442275 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:30.442257 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73\": container with ID starting with b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73 not found: ID does not exist" containerID="b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73" Nov 01 23:10:30.442333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.442283 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73"} err="failed to get container status \"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73\": rpc error: code = NotFound desc = could not find container \"b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73\": container with ID starting with b862802f1949bed19d4f268db94d3e1e7b2be6b209edbddaee46d2ff43413f73 not found: ID does not exist" Nov 01 23:10:30.454565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.454541 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:10:30.458639 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.458617 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks"] Nov 01 23:10:30.715948 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:30.713945 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/premium-w2-9ad303-kserve-d8668d79d-lb8d9" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:10:32.373995 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:32.373963 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" path="/var/lib/kubelet/pods/9cce4af7-41f1-46ef-a412-08a0203f4c5f/volumes" Nov 01 23:10:32.374490 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:32.374475 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c478da01-6384-454f-b3ce-e521652a51ff" path="/var/lib/kubelet/pods/c478da01-6384-454f-b3ce-e521652a51ff/volumes" Nov 01 23:10:32.758429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:32.758383 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec0da7-5dc9f49d67-2twks" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.178:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:10:46.573364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:46.572439 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:10:46.573364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:46.572757 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" containerID="cri-o://b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e" gracePeriod=30 Nov 01 23:10:47.890715 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:47.890680 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 23:10:47.891248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:47.890931 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" containerID="cri-o://4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a" gracePeriod=30 Nov 01 23:10:48.777918 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:48.777881 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:10:51.902571 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:51.902544 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054270 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054403 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054459 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054504 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054543 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054665 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054706 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054816 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.054875 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.054873 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\") pod \"405525e1-ba54-4275-b2f1-7c31e8f88f82\" (UID: \"405525e1-ba54-4275-b2f1-7c31e8f88f82\") " Nov 01 23:10:52.089000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.058093 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.089000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.059777 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:52.091715 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.090879 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.101026 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.100964 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.101169 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.101030 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "kube-api-access-m6j6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.103936 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.103905 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.115904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.114042 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:52.115904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.114047 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.115904 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.114196 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token" pod "405525e1-ba54-4275-b2f1-7c31e8f88f82" (UID: "405525e1-ba54-4275-b2f1-7c31e8f88f82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.185665 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.185623 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.185954 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.185931 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186066 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186186 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6j6m\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-kube-api-access-m6j6m\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186278 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405525e1-ba54-4275-b2f1-7c31e8f88f82-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186379 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186480 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186468 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186573 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186561 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405525e1-ba54-4275-b2f1-7c31e8f88f82-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.186657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.186645 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405525e1-ba54-4275-b2f1-7c31e8f88f82-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.641778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.641742 5778 generic.go:356] "Generic (PLEG): container finished" podID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerID="b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e" exitCode=0 Nov 01 23:10:52.641989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.641807 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" Nov 01 23:10:52.641989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.641822 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerDied","Data":"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e"} Nov 01 23:10:52.641989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.641931 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z" event={"ID":"405525e1-ba54-4275-b2f1-7c31e8f88f82","Type":"ContainerDied","Data":"8dd2dd7ec4d8a40e5efb832772a630bf2687604adbbc4ead114853f9f7fe5a61"} Nov 01 23:10:52.641989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.641966 5778 scope.go:122] "RemoveContainer" containerID="b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e" Nov 01 23:10:52.655390 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.655365 5778 scope.go:122] "RemoveContainer" containerID="5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9" Nov 01 23:10:52.674422 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.674389 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:10:52.675833 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.675807 5778 scope.go:122] "RemoveContainer" containerID="b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e" Nov 01 23:10:52.676228 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:52.676196 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e\": container with ID starting with b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e not found: ID does not exist" containerID="b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e" Nov 01 23:10:52.676325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.676237 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e"} err="failed to get container status \"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e\": rpc error: code = NotFound desc = could not find container \"b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e\": container with ID starting with b7620962a329df54cbbd74c92331a061cf9bee2e270b52f700cae48dd8ae0e0e not found: ID does not exist" Nov 01 23:10:52.676325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.676256 5778 scope.go:122] "RemoveContainer" containerID="5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9" Nov 01 23:10:52.676563 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:52.676546 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9\": container with ID starting with 5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9 not found: ID does not exist" containerID="5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9" Nov 01 23:10:52.676613 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.676569 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9"} err="failed to get container status \"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9\": rpc error: code = NotFound desc = could not find container \"5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9\": container with ID starting with 5075fd3f4a4052db657f7eae3591e84c531928910f2175be000b04a72b7815d9 not found: ID does not exist" Nov 01 23:10:52.679168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:52.679145 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-xtd5z"] Nov 01 23:10:53.265391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.265364 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 23:10:53.300186 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300151 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300272 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300310 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300394 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300431 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300445 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.300501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300464 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300517 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300615 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.300727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300627 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.300727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300723 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\") pod \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\" (UID: \"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285\") " Nov 01 23:10:53.301008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.300816 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.301244 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.301219 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.301332 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.301250 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.301332 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.301267 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.301443 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.301320 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.301617 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.301583 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:53.306298 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.306256 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "kube-api-access-4qch9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:53.306565 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.306538 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:53.306781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.306744 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:53.314187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.314165 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy" pod "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" (UID: "ecf7feda-0a0f-4cbe-ac5e-cdbf32902285"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.402319 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402279 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.402319 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402320 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.402594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402368 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.402594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402386 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qch9\" (UniqueName: \"kubernetes.io/projected/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-kube-api-access-4qch9\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.402594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402400 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.402594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.402413 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.652016 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.651977 5778 generic.go:356] "Generic (PLEG): container finished" podID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerID="4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a" exitCode=0 Nov 01 23:10:53.652224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.652062 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" Nov 01 23:10:53.652224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.652069 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" event={"ID":"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285","Type":"ContainerDied","Data":"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a"} Nov 01 23:10:53.652224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.652153 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk" event={"ID":"ecf7feda-0a0f-4cbe-ac5e-cdbf32902285","Type":"ContainerDied","Data":"b2a68eb4c5bef1ef7173175d97a2743fbdcbb703e94bc9d3d42b5e7694a0a4f3"} Nov 01 23:10:53.652224 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.652187 5778 scope.go:122] "RemoveContainer" containerID="4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a" Nov 01 23:10:53.664544 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.664527 5778 scope.go:122] "RemoveContainer" containerID="4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a" Nov 01 23:10:53.664935 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:53.664900 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a\": container with ID starting with 4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a not found: ID does not exist" containerID="4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a" Nov 01 23:10:53.665065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.664942 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a"} err="failed to get container status \"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a\": rpc error: code = NotFound desc = could not find container \"4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a\": container with ID starting with 4ab7b590858bc1cc5e58ca9400b442d41cb1b99805bb60da7ef53ce90ee01a9a not found: ID does not exist" Nov 01 23:10:53.689928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.689889 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 23:10:53.700685 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:53.700654 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-tmkvk"] Nov 01 23:10:54.375668 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:54.375631 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" path="/var/lib/kubelet/pods/405525e1-ba54-4275-b2f1-7c31e8f88f82/volumes" Nov 01 23:10:54.376320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:54.376295 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" path="/var/lib/kubelet/pods/ecf7feda-0a0f-4cbe-ac5e-cdbf32902285/volumes" Nov 01 23:10:58.679520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:58.679449 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 23:10:58.680114 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:58.679746 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" podUID="1ddf4019-13be-4213-937a-78ef85850d51" containerName="maas-api" containerID="cri-o://3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1" gracePeriod=30 Nov 01 23:10:59.280455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.280427 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 23:10:59.401657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.401620 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\") pod \"1ddf4019-13be-4213-937a-78ef85850d51\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " Nov 01 23:10:59.401842 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.401702 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\") pod \"1ddf4019-13be-4213-937a-78ef85850d51\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " Nov 01 23:10:59.401842 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.401758 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\" (UniqueName: \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\") pod \"1ddf4019-13be-4213-937a-78ef85850d51\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " Nov 01 23:10:59.401842 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.401788 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\") pod \"1ddf4019-13be-4213-937a-78ef85850d51\" (UID: \"1ddf4019-13be-4213-937a-78ef85850d51\") " Nov 01 23:10:59.402076 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.402047 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca" pod "1ddf4019-13be-4213-937a-78ef85850d51" (UID: "1ddf4019-13be-4213-937a-78ef85850d51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:59.402541 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.402465 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddf4019-13be-4213-937a-78ef85850d51-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:59.404203 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.404181 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs" pod "1ddf4019-13be-4213-937a-78ef85850d51" (UID: "1ddf4019-13be-4213-937a-78ef85850d51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:59.404323 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.404201 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls" pod "1ddf4019-13be-4213-937a-78ef85850d51" (UID: "1ddf4019-13be-4213-937a-78ef85850d51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:59.404363 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.404331 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5" pod "1ddf4019-13be-4213-937a-78ef85850d51" (UID: "1ddf4019-13be-4213-937a-78ef85850d51"). InnerVolumeSpecName "kube-api-access-dptq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:59.504276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.504240 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:59.504276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.504270 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptq5\" (UniqueName: \"kubernetes.io/projected/1ddf4019-13be-4213-937a-78ef85850d51-kube-api-access-dptq5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:59.504276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.504283 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddf4019-13be-4213-937a-78ef85850d51-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:59.722896 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.722844 5778 generic.go:356] "Generic (PLEG): container finished" podID="1ddf4019-13be-4213-937a-78ef85850d51" containerID="3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1" exitCode=0 Nov 01 23:10:59.723402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.722936 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" Nov 01 23:10:59.723402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.722943 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" event={"ID":"1ddf4019-13be-4213-937a-78ef85850d51","Type":"ContainerDied","Data":"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1"} Nov 01 23:10:59.723402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.723001 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n" event={"ID":"1ddf4019-13be-4213-937a-78ef85850d51","Type":"ContainerDied","Data":"14a4f37c508a72078627209bd514f036b833df59b065fad79c603cde1fbdea97"} Nov 01 23:10:59.723402 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.723027 5778 scope.go:122] "RemoveContainer" containerID="3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1" Nov 01 23:10:59.736404 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.736379 5778 scope.go:122] "RemoveContainer" containerID="3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1" Nov 01 23:10:59.736740 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:10:59.736715 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1\": container with ID starting with 3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1 not found: ID does not exist" containerID="3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1" Nov 01 23:10:59.736837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.736749 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1"} err="failed to get container status \"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1\": rpc error: code = NotFound desc = could not find container \"3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1\": container with ID starting with 3aebe378fd69326d5d4264a6f10df1dfe7a4221e17bd2502c317f52e055b3ef1 not found: ID does not exist" Nov 01 23:10:59.748416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.748390 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 23:10:59.753883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:10:59.753842 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c6cec6-6d6df9c77-zvs8n"] Nov 01 23:11:00.375537 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:00.375504 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddf4019-13be-4213-937a-78ef85850d51" path="/var/lib/kubelet/pods/1ddf4019-13be-4213-937a-78ef85850d51/volumes" Nov 01 23:11:02.659182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:02.657565 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:11:02.659182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:02.657801 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" podUID="d448a01a-e368-4968-8912-9d8b1cb19f8f" containerName="maas-api" containerID="cri-o://ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11" gracePeriod=30 Nov 01 23:11:03.076123 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.076090 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.178509 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\" (UniqueName: \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\") pod \"d448a01a-e368-4968-8912-9d8b1cb19f8f\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.179375 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\") pod \"d448a01a-e368-4968-8912-9d8b1cb19f8f\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.179886 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\") pod \"d448a01a-e368-4968-8912-9d8b1cb19f8f\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.180322 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\") pod \"d448a01a-e368-4968-8912-9d8b1cb19f8f\" (UID: \"d448a01a-e368-4968-8912-9d8b1cb19f8f\") " Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.180802 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca" pod "d448a01a-e368-4968-8912-9d8b1cb19f8f" (UID: "d448a01a-e368-4968-8912-9d8b1cb19f8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:03.183900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.181415 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448a01a-e368-4968-8912-9d8b1cb19f8f-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.189428 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.189398 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls" pod "d448a01a-e368-4968-8912-9d8b1cb19f8f" (UID: "d448a01a-e368-4968-8912-9d8b1cb19f8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:03.189603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.189583 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w" pod "d448a01a-e368-4968-8912-9d8b1cb19f8f" (UID: "d448a01a-e368-4968-8912-9d8b1cb19f8f"). InnerVolumeSpecName "kube-api-access-mm65w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:03.189755 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.189724 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs" pod "d448a01a-e368-4968-8912-9d8b1cb19f8f" (UID: "d448a01a-e368-4968-8912-9d8b1cb19f8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:03.282757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.282647 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.282757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.282693 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448a01a-e368-4968-8912-9d8b1cb19f8f-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.282757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.282710 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm65w\" (UniqueName: \"kubernetes.io/projected/d448a01a-e368-4968-8912-9d8b1cb19f8f-kube-api-access-mm65w\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.791046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.791010 5778 generic.go:356] "Generic (PLEG): container finished" podID="d448a01a-e368-4968-8912-9d8b1cb19f8f" containerID="ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11" exitCode=0 Nov 01 23:11:03.791486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.791075 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" Nov 01 23:11:03.791486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.791096 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" event={"ID":"d448a01a-e368-4968-8912-9d8b1cb19f8f","Type":"ContainerDied","Data":"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11"} Nov 01 23:11:03.791486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.791178 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll" event={"ID":"d448a01a-e368-4968-8912-9d8b1cb19f8f","Type":"ContainerDied","Data":"d9d26441c36e737d0b1fee18b340285bda6bf4b7e3aa9c97dcd8756aee31a616"} Nov 01 23:11:03.791486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.791205 5778 scope.go:122] "RemoveContainer" containerID="ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11" Nov 01 23:11:03.802940 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.802919 5778 scope.go:122] "RemoveContainer" containerID="ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11" Nov 01 23:11:03.803220 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:03.803200 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11\": container with ID starting with ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11 not found: ID does not exist" containerID="ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11" Nov 01 23:11:03.803294 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.803232 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11"} err="failed to get container status \"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11\": rpc error: code = NotFound desc = could not find container \"ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11\": container with ID starting with ab0caf5356edff01612e86f1d18baf3accb06d12d9a7a83aa6a30f018ad63a11 not found: ID does not exist" Nov 01 23:11:03.822189 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.822163 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:11:03.826274 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:03.826139 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-46ce02-56fc8dc887-htdll"] Nov 01 23:11:04.376629 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:04.376588 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d448a01a-e368-4968-8912-9d8b1cb19f8f" path="/var/lib/kubelet/pods/d448a01a-e368-4968-8912-9d8b1cb19f8f/volumes" Nov 01 23:11:12.405876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:12.405579 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:11:17.045920 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:17.044719 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:11:18.776722 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:18.776679 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:11:27.855876 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:27.855819 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 23:11:27.856863 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:27.856811 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" podUID="9eae91fb-897c-4288-ac57-86173fe4e051" containerName="maas-api" containerID="cri-o://dd13864b69cf8aff6475301f9c9401313ec3a3ef0974722bd6715e0f5d95646c" gracePeriod=30 Nov 01 23:11:28.011384 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.011340 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" containerID="cri-o://22a420fd2b66821c76a15c45bb3d221e76ce7968b1eabd62a7df490e17213946" gracePeriod=45 Nov 01 23:11:28.244883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.238072 5778 generic.go:356] "Generic (PLEG): container finished" podID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerID="22a420fd2b66821c76a15c45bb3d221e76ce7968b1eabd62a7df490e17213946" exitCode=0 Nov 01 23:11:28.266541 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.248965 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" event={"ID":"6a9d05f7-c83d-4432-a138-5070de7f3424","Type":"ContainerDied","Data":"22a420fd2b66821c76a15c45bb3d221e76ce7968b1eabd62a7df490e17213946"} Nov 01 23:11:28.272638 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.272444 5778 generic.go:356] "Generic (PLEG): container finished" podID="9eae91fb-897c-4288-ac57-86173fe4e051" containerID="dd13864b69cf8aff6475301f9c9401313ec3a3ef0974722bd6715e0f5d95646c" exitCode=0 Nov 01 23:11:28.272638 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.272560 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" event={"ID":"9eae91fb-897c-4288-ac57-86173fe4e051","Type":"ContainerDied","Data":"dd13864b69cf8aff6475301f9c9401313ec3a3ef0974722bd6715e0f5d95646c"} Nov 01 23:11:28.351219 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.351186 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 23:11:28.403013 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.402970 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\") pod \"9eae91fb-897c-4288-ac57-86173fe4e051\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " Nov 01 23:11:28.403205 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.403075 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\" (UniqueName: \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\") pod \"9eae91fb-897c-4288-ac57-86173fe4e051\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " Nov 01 23:11:28.403205 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.403168 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") pod \"9eae91fb-897c-4288-ac57-86173fe4e051\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " Nov 01 23:11:28.403337 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.403278 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\") pod \"9eae91fb-897c-4288-ac57-86173fe4e051\" (UID: \"9eae91fb-897c-4288-ac57-86173fe4e051\") " Nov 01 23:11:28.403910 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.403876 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca" pod "9eae91fb-897c-4288-ac57-86173fe4e051" (UID: "9eae91fb-897c-4288-ac57-86173fe4e051"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:28.404686 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.404362 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eae91fb-897c-4288-ac57-86173fe4e051-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.409462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.409403 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr" pod "9eae91fb-897c-4288-ac57-86173fe4e051" (UID: "9eae91fb-897c-4288-ac57-86173fe4e051"). InnerVolumeSpecName "kube-api-access-82vpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:28.410127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.410097 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls" pod "9eae91fb-897c-4288-ac57-86173fe4e051" (UID: "9eae91fb-897c-4288-ac57-86173fe4e051"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:28.410368 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.410344 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs" pod "9eae91fb-897c-4288-ac57-86173fe4e051" (UID: "9eae91fb-897c-4288-ac57-86173fe4e051"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:28.458000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.457974 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:11:28.510653 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.510611 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.511018 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.510991 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.511257 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.511236 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.512785 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.512749 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.513939 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.513867 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vpr\" (UniqueName: \"kubernetes.io/projected/9eae91fb-897c-4288-ac57-86173fe4e051-kube-api-access-82vpr\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.514086 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.514071 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eae91fb-897c-4288-ac57-86173fe4e051-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.514209 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.512984 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:28.514655 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.514634 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:28.523363 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.523323 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:28.614949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.614901 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.614949 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.614952 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\" (UniqueName: \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.615225 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615014 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\") pod \"6a9d05f7-c83d-4432-a138-5070de7f3424\" (UID: \"6a9d05f7-c83d-4432-a138-5070de7f3424\") " Nov 01 23:11:28.615329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615303 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:28.615724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615625 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.615724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615651 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9d05f7-c83d-4432-a138-5070de7f3424-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.615724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615667 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.615724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.615680 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.617406 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.617383 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:28.617501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.617457 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd" pod "6a9d05f7-c83d-4432-a138-5070de7f3424" (UID: "6a9d05f7-c83d-4432-a138-5070de7f3424"). InnerVolumeSpecName "kube-api-access-ctqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:28.718680 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.718591 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9d05f7-c83d-4432-a138-5070de7f3424-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:28.718680 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:28.718641 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqmd\" (UniqueName: \"kubernetes.io/projected/6a9d05f7-c83d-4432-a138-5070de7f3424-kube-api-access-ctqmd\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:29.302922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.302887 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" Nov 01 23:11:29.302922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.302908 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47" event={"ID":"6a9d05f7-c83d-4432-a138-5070de7f3424","Type":"ContainerDied","Data":"98f18ac02ab73a28be09594e22674fa766d76b3636b7742b3275910d7c1ca377"} Nov 01 23:11:29.303555 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.302972 5778 scope.go:122] "RemoveContainer" containerID="22a420fd2b66821c76a15c45bb3d221e76ce7968b1eabd62a7df490e17213946" Nov 01 23:11:29.306379 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.306344 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" event={"ID":"9eae91fb-897c-4288-ac57-86173fe4e051","Type":"ContainerDied","Data":"43deae3d2ef781931bb0422bdcbf14f2cea876f26944b3140384de95dde98b43"} Nov 01 23:11:29.306506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.306458 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc" Nov 01 23:11:29.316685 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.316230 5778 scope.go:122] "RemoveContainer" containerID="dd13864b69cf8aff6475301f9c9401313ec3a3ef0974722bd6715e0f5d95646c" Nov 01 23:11:29.356588 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.356556 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:11:29.371142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.371112 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-7366bf/sub-test-model-7366bf-kserve-645d9b97bf-44b47"] Nov 01 23:11:29.397221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.397182 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 23:11:29.403667 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.403643 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7366bf-6d5f68867f-5hbrc"] Nov 01 23:11:29.860568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.860522 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 23:11:29.860862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:29.860798 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" containerID="cri-o://8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2" gracePeriod=30 Nov 01 23:11:30.373868 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:30.373824 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" path="/var/lib/kubelet/pods/6a9d05f7-c83d-4432-a138-5070de7f3424/volumes" Nov 01 23:11:30.374365 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:30.374347 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eae91fb-897c-4288-ac57-86173fe4e051" path="/var/lib/kubelet/pods/9eae91fb-897c-4288-ac57-86173fe4e051/volumes" Nov 01 23:11:32.075530 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.075484 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" containerName="main" containerID="cri-o://b5e73513794edc4744c5331e0dd495b570de11649364c1bdfeb09a7df7ff6fac" gracePeriod=45 Nov 01 23:11:32.336588 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.336487 5778 generic.go:356] "Generic (PLEG): container finished" podID="bf7ae301-5191-4511-912d-fb6bdb280530" containerID="b5e73513794edc4744c5331e0dd495b570de11649364c1bdfeb09a7df7ff6fac" exitCode=0 Nov 01 23:11:32.336781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.336620 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" event={"ID":"bf7ae301-5191-4511-912d-fb6bdb280530","Type":"ContainerDied","Data":"b5e73513794edc4744c5331e0dd495b570de11649364c1bdfeb09a7df7ff6fac"} Nov 01 23:11:32.385484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.385455 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:11:32.571345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.571280 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.571629 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.571609 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\" (UniqueName: \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.571744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.571730 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.571928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.571913 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.572116 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.572102 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.572333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.572319 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\") pod \"bf7ae301-5191-4511-912d-fb6bdb280530\" (UID: \"bf7ae301-5191-4511-912d-fb6bdb280530\") " Nov 01 23:11:32.576868 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.576350 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:32.576868 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.576762 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:32.577757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.577162 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:32.583494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.583466 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:32.584291 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.584268 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "kube-api-access-hk7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:32.590577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.590489 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs" pod "bf7ae301-5191-4511-912d-fb6bdb280530" (UID: "bf7ae301-5191-4511-912d-fb6bdb280530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:32.676417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676372 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7ae301-5191-4511-912d-fb6bdb280530-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:32.676417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676414 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7xc\" (UniqueName: \"kubernetes.io/projected/bf7ae301-5191-4511-912d-fb6bdb280530-kube-api-access-hk7xc\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:32.676808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676433 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:32.676808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676448 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:32.676808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676493 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:32.676808 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:32.676506 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7ae301-5191-4511-912d-fb6bdb280530-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:33.347330 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:33.347302 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" Nov 01 23:11:33.347330 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:33.347310 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw" event={"ID":"bf7ae301-5191-4511-912d-fb6bdb280530","Type":"ContainerDied","Data":"c4317dddc9c556f58217df806bb47589b3204aaad92644d6c496e7b6258487ac"} Nov 01 23:11:33.347994 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:33.347359 5778 scope.go:122] "RemoveContainer" containerID="b5e73513794edc4744c5331e0dd495b570de11649364c1bdfeb09a7df7ff6fac" Nov 01 23:11:33.370651 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:33.370621 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:11:33.373330 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:33.373285 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-075771/sub-test-model-075771-kserve-659c4cdfdc-j4llw"] Nov 01 23:11:34.376199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:34.376160 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" path="/var/lib/kubelet/pods/bf7ae301-5191-4511-912d-fb6bdb280530/volumes" Nov 01 23:11:35.156700 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.156664 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 23:11:35.218741 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.218704 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.218961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.218784 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.218961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.218815 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.218961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.218919 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219149 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219001 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219149 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219067 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219149 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219065 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.219333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219208 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219262 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219457 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219371 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\") pod \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\" (UID: \"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba\") " Nov 01 23:11:35.219524 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219497 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.219579 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219567 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:35.219748 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219727 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.219980 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219958 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.220260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.219985 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.220260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.220000 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.220260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.220015 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.220441 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.220419 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.222099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.222030 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.222099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.222034 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:35.222416 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.222388 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "kube-api-access-jslck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:35.223518 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.223498 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo" pod "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" (UID: "de5e8124-afe2-4e38-b3e1-4e63cb2a7aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:35.260596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.260552 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 23:11:35.320769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.320730 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.320769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.320765 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.320769 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.320775 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslck\" (UniqueName: \"kubernetes.io/projected/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-kube-api-access-jslck\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.321120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.320785 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.321120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.320794 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.366611 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.366576 5778 generic.go:356] "Generic (PLEG): container finished" podID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerID="8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2" exitCode=0 Nov 01 23:11:35.366816 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.366626 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" Nov 01 23:11:35.366816 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.366657 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" event={"ID":"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba","Type":"ContainerDied","Data":"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2"} Nov 01 23:11:35.366816 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.366714 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld" event={"ID":"de5e8124-afe2-4e38-b3e1-4e63cb2a7aba","Type":"ContainerDied","Data":"cc928890fbffb86babef82cae46788bc1ddb58bfa79bb9ace9e99dea6abf0a8a"} Nov 01 23:11:35.366816 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.366745 5778 scope.go:122] "RemoveContainer" containerID="8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2" Nov 01 23:11:35.382446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.382419 5778 scope.go:122] "RemoveContainer" containerID="8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2" Nov 01 23:11:35.382896 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:35.382867 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2\": container with ID starting with 8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2 not found: ID does not exist" containerID="8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2" Nov 01 23:11:35.382970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.382907 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2"} err="failed to get container status \"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2\": rpc error: code = NotFound desc = could not find container \"8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2\": container with ID starting with 8e4a36394fdc652bb6dcfb0c1d010bcc54d03658eaefdcf7cbba6c34baa71ac2 not found: ID does not exist" Nov 01 23:11:35.434008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.430827 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 23:11:35.434008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:35.433122 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-hxkld"] Nov 01 23:11:36.377088 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:36.377050 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" path="/var/lib/kubelet/pods/de5e8124-afe2-4e38-b3e1-4e63cb2a7aba/volumes" Nov 01 23:11:36.739877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:36.737590 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 23:11:36.739877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:36.737845 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" containerID="cri-o://ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde" gracePeriod=30 Nov 01 23:11:37.127977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:37.125193 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 23:11:37.127977 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:37.125421 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" containerID="cri-o://5cf43a6f7933e45cb86cc93ff35713ccb65b111e234eb5778bba4ee351b4e5c9" gracePeriod=30 Nov 01 23:11:37.564459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:37.564420 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:11:37.564654 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:37.564485 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:11:38.366673 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:38.366639 5778 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:11:38.367177 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:38.366712 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:11:39.936998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:39.936956 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:11:40.630320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:40.630280 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 23:11:42.078490 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.078465 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 23:11:42.167931 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.167902 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168096 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.167958 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168096 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168085 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168220 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168159 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168220 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168182 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.168220 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168194 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168265 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.168376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168291 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168329 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168347 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.168594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168409 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168439 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\") pod \"e7520856-7bc0-4cce-a430-f537fe2e800d\" (UID: \"e7520856-7bc0-4cce-a430-f537fe2e800d\") " Nov 01 23:11:42.168594 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168542 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.168772 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168706 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.169012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168934 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.169012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168954 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.169012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168968 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520856-7bc0-4cce-a430-f537fe2e800d-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.169012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168981 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.169012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.168990 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.179184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.179163 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.189225 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.189204 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "kube-api-access-6gqpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.189321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.189217 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.189321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.189227 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token" pod "e7520856-7bc0-4cce-a430-f537fe2e800d" (UID: "e7520856-7bc0-4cce-a430-f537fe2e800d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.270758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.270706 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.270758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.270736 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.270758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.270746 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqpm\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-kube-api-access-6gqpm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.270758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.270757 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520856-7bc0-4cce-a430-f537fe2e800d-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.422910 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.422799 5778 generic.go:356] "Generic (PLEG): container finished" podID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerID="ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde" exitCode=0 Nov 01 23:11:42.422910 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.422878 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" Nov 01 23:11:42.423093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.422840 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerDied","Data":"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde"} Nov 01 23:11:42.423093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.422998 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774" event={"ID":"e7520856-7bc0-4cce-a430-f537fe2e800d","Type":"ContainerDied","Data":"3fd9a7325f8589a7a55883b2783b23e340fc92f110f05d2f5497eab35d7da7b9"} Nov 01 23:11:42.423093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.423028 5778 scope.go:122] "RemoveContainer" containerID="ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde" Nov 01 23:11:42.426252 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.426223 5778 generic.go:356] "Generic (PLEG): container finished" podID="3d157576-5447-4574-8903-4cc537809016" containerID="5cf43a6f7933e45cb86cc93ff35713ccb65b111e234eb5778bba4ee351b4e5c9" exitCode=0 Nov 01 23:11:42.427138 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.426283 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerDied","Data":"5cf43a6f7933e45cb86cc93ff35713ccb65b111e234eb5778bba4ee351b4e5c9"} Nov 01 23:11:42.428083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.428056 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:11:42.437538 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.437523 5778 scope.go:122] "RemoveContainer" containerID="37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6" Nov 01 23:11:42.450259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.450226 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 23:11:42.459337 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.459316 5778 scope.go:122] "RemoveContainer" containerID="ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde" Nov 01 23:11:42.459769 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:42.459745 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde\": container with ID starting with ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde not found: ID does not exist" containerID="ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde" Nov 01 23:11:42.459944 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.459782 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde"} err="failed to get container status \"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde\": rpc error: code = NotFound desc = could not find container \"ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde\": container with ID starting with ff959ab5c63a048108fa28eccaf53628199251644743b478016429828b754cde not found: ID does not exist" Nov 01 23:11:42.459944 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.459821 5778 scope.go:122] "RemoveContainer" containerID="37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6" Nov 01 23:11:42.460273 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:42.460215 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6\": container with ID starting with 37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6 not found: ID does not exist" containerID="37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6" Nov 01 23:11:42.460354 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.460283 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6"} err="failed to get container status \"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6\": rpc error: code = NotFound desc = could not find container \"37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6\": container with ID starting with 37b85f416ef37b58f47b2628b64e7dce3e89aa1362f633ef5780fd06589171e6 not found: ID does not exist" Nov 01 23:11:42.460354 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.460306 5778 scope.go:122] "RemoveContainer" containerID="b85aceedd9d9cd5428d5e60f80115a7a2bb50e2460c0590c8d8ff620c08bb078" Nov 01 23:11:42.461270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.461109 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-k6774"] Nov 01 23:11:42.574380 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574283 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574380 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574339 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574387 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574420 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574443 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574512 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574584 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.574951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574639 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574666 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.574951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574681 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574831 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\") pod \"3d157576-5447-4574-8903-4cc537809016\" (UID: \"3d157576-5447-4574-8903-4cc537809016\") " Nov 01 23:11:42.574951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.574912 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.575213 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575010 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.575213 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575185 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.575688 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575663 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.575756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575694 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.575756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575710 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.575756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575725 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d157576-5447-4574-8903-4cc537809016-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.575756 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.575740 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.577237 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.577217 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "kube-api-access-9vrhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.577304 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.577244 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.577514 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.577495 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.577569 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.577520 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy" pod "3d157576-5447-4574-8903-4cc537809016" (UID: "3d157576-5447-4574-8903-4cc537809016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.676752 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.676715 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrhw\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-kube-api-access-9vrhw\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.676752 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.676747 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d157576-5447-4574-8903-4cc537809016-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.676752 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.676757 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d157576-5447-4574-8903-4cc537809016-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.677100 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:42.676766 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d157576-5447-4574-8903-4cc537809016-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:43.437674 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:43.437644 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" Nov 01 23:11:43.438175 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:43.437639 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8" event={"ID":"3d157576-5447-4574-8903-4cc537809016","Type":"ContainerDied","Data":"ea1b888809b11b4e17b7682338662aeb01ceb9299a983d17dd911ab363044b5a"} Nov 01 23:11:43.438175 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:43.437819 5778 scope.go:122] "RemoveContainer" containerID="5cf43a6f7933e45cb86cc93ff35713ccb65b111e234eb5778bba4ee351b4e5c9" Nov 01 23:11:43.493575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:43.493545 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 23:11:43.499754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:43.499723 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-cw2v8"] Nov 01 23:11:44.376258 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:44.376218 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d157576-5447-4574-8903-4cc537809016" path="/var/lib/kubelet/pods/3d157576-5447-4574-8903-4cc537809016/volumes" Nov 01 23:11:44.376960 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:44.376937 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" path="/var/lib/kubelet/pods/e7520856-7bc0-4cce-a430-f537fe2e800d/volumes" Nov 01 23:11:44.498884 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:44.496943 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:11:45.631713 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:45.631665 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" podUID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" containerName="payload-pre-processing" containerID="cri-o://596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49" gracePeriod=25 Nov 01 23:11:45.941589 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:45.941562 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 23:11:46.030374 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.030343 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\" (UniqueName: \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\") pod \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " Nov 01 23:11:46.030556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.030417 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\") pod \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\" (UID: \"ce72afa2-7f2e-43f2-98fa-6d38973ada6b\") " Nov 01 23:11:46.030828 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.030804 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume" pod "ce72afa2-7f2e-43f2-98fa-6d38973ada6b" (UID: "ce72afa2-7f2e-43f2-98fa-6d38973ada6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:46.032897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.032872 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7" pod "ce72afa2-7f2e-43f2-98fa-6d38973ada6b" (UID: "ce72afa2-7f2e-43f2-98fa-6d38973ada6b"). InnerVolumeSpecName "kube-api-access-84ks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:46.132321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.132281 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84ks7\" (UniqueName: \"kubernetes.io/projected/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-kube-api-access-84ks7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.132321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.132324 5778 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce72afa2-7f2e-43f2-98fa-6d38973ada6b-plugins-config-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.461822 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.461785 5778 generic.go:356] "Generic (PLEG): container finished" podID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" containerID="596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49" exitCode=0 Nov 01 23:11:46.462035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.461888 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" Nov 01 23:11:46.462035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.461879 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" event={"ID":"ce72afa2-7f2e-43f2-98fa-6d38973ada6b","Type":"ContainerDied","Data":"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49"} Nov 01 23:11:46.462035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.462020 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd" event={"ID":"ce72afa2-7f2e-43f2-98fa-6d38973ada6b","Type":"ContainerDied","Data":"5f4d8ec9fdc03748b5dca7f55157c77f9da8df6df89834273374ec8cce37bd1f"} Nov 01 23:11:46.462239 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.462054 5778 scope.go:122] "RemoveContainer" containerID="596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49" Nov 01 23:11:46.472793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.472733 5778 scope.go:122] "RemoveContainer" containerID="596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49" Nov 01 23:11:46.473118 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:46.473083 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49\": container with ID starting with 596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49 not found: ID does not exist" containerID="596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49" Nov 01 23:11:46.473201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.473120 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49"} err="failed to get container status \"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49\": rpc error: code = NotFound desc = could not find container \"596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49\": container with ID starting with 596d7eeba422d21501afd71c8b425da831213ab0933e0c46f9353d456d9f6c49 not found: ID does not exist" Nov 01 23:11:46.483972 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.483950 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 23:11:46.489632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:46.489610 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f338e4-6c59bf4659-c8vsd"] Nov 01 23:11:48.380420 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:48.380375 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" path="/var/lib/kubelet/pods/ce72afa2-7f2e-43f2-98fa-6d38973ada6b/volumes" Nov 01 23:11:48.774295 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:48.774258 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:11:50.283966 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.283895 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" containerID="cri-o://b9994d9ec8785b7dcba1d73a94336b556bf47cc7a346f8b5bb08a43f2dc97301" gracePeriod=45 Nov 01 23:11:50.500346 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.500299 5778 generic.go:356] "Generic (PLEG): container finished" podID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerID="b9994d9ec8785b7dcba1d73a94336b556bf47cc7a346f8b5bb08a43f2dc97301" exitCode=0 Nov 01 23:11:50.500511 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.500378 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" event={"ID":"96b1296e-ee30-4ccb-8f99-96a3552178c9","Type":"ContainerDied","Data":"b9994d9ec8785b7dcba1d73a94336b556bf47cc7a346f8b5bb08a43f2dc97301"} Nov 01 23:11:50.589604 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.589576 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 23:11:50.598184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598154 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.598361 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598238 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.598435 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598358 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\" (UniqueName: \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.598435 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598403 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.598435 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598430 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.598616 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.598505 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\") pod \"96b1296e-ee30-4ccb-8f99-96a3552178c9\" (UID: \"96b1296e-ee30-4ccb-8f99-96a3552178c9\") " Nov 01 23:11:50.599139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.599049 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.599422 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.599182 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.599680 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.599652 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.602313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.602255 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "kube-api-access-rhqqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:50.603331 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.602588 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:50.633107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.615191 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm" pod "96b1296e-ee30-4ccb-8f99-96a3552178c9" (UID: "96b1296e-ee30-4ccb-8f99-96a3552178c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.700395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700360 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.700395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700392 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b1296e-ee30-4ccb-8f99-96a3552178c9-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.700657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700404 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqqg\" (UniqueName: \"kubernetes.io/projected/96b1296e-ee30-4ccb-8f99-96a3552178c9-kube-api-access-rhqqg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.700657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700415 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.700657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700423 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.700657 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:50.700431 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b1296e-ee30-4ccb-8f99-96a3552178c9-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:51.468005 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.467941 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:11:51.508814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.508785 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" Nov 01 23:11:51.509035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.508818 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72" event={"ID":"96b1296e-ee30-4ccb-8f99-96a3552178c9","Type":"ContainerDied","Data":"500fea8c9c534070afd9e619a2cfcdda1868edff13dc47635f3d456d5a4005fb"} Nov 01 23:11:51.509035 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.508881 5778 scope.go:122] "RemoveContainer" containerID="b9994d9ec8785b7dcba1d73a94336b556bf47cc7a346f8b5bb08a43f2dc97301" Nov 01 23:11:51.538574 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.538546 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 23:11:51.542841 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:51.542808 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-w1-f338e4-kserve-86f5dd5869-8xn72"] Nov 01 23:11:52.375572 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:52.375539 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" path="/var/lib/kubelet/pods/96b1296e-ee30-4ccb-8f99-96a3552178c9/volumes" Nov 01 23:11:54.961498 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:54.961420 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" podUID="5b12473d-3d7d-46fb-ae16-569af651e5da" containerName="main" containerID="cri-o://f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738" gracePeriod=45 Nov 01 23:11:55.327154 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.327116 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:11:55.370754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.370711 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.370998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.370790 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.370998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.370819 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.370998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.370985 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.371122 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.371043 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\" (UniqueName: \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.371122 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.371107 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\") pod \"5b12473d-3d7d-46fb-ae16-569af651e5da\" (UID: \"5b12473d-3d7d-46fb-ae16-569af651e5da\") " Nov 01 23:11:55.371734 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.371708 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:55.372351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.372319 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:55.372546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.372526 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:55.374881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.374814 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:55.375024 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.374974 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:55.376639 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.376619 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w" pod "5b12473d-3d7d-46fb-ae16-569af651e5da" (UID: "5b12473d-3d7d-46fb-ae16-569af651e5da"). InnerVolumeSpecName "kube-api-access-9cw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:55.472613 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472571 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.472613 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472610 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cw2w\" (UniqueName: \"kubernetes.io/projected/5b12473d-3d7d-46fb-ae16-569af651e5da-kube-api-access-9cw2w\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.472928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472626 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.472928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472636 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b12473d-3d7d-46fb-ae16-569af651e5da-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.472928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472645 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.472928 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.472656 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b12473d-3d7d-46fb-ae16-569af651e5da-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:55.572257 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.572170 5778 generic.go:356] "Generic (PLEG): container finished" podID="5b12473d-3d7d-46fb-ae16-569af651e5da" containerID="f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738" exitCode=0 Nov 01 23:11:55.572257 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.572218 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" Nov 01 23:11:55.572462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.572266 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" event={"ID":"5b12473d-3d7d-46fb-ae16-569af651e5da","Type":"ContainerDied","Data":"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738"} Nov 01 23:11:55.572462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.572317 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj" event={"ID":"5b12473d-3d7d-46fb-ae16-569af651e5da","Type":"ContainerDied","Data":"758349a4602f1cec7f790ee3d30a8d856730399bb0d713ba49339d3f36d5e80e"} Nov 01 23:11:55.572462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.572340 5778 scope.go:122] "RemoveContainer" containerID="f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738" Nov 01 23:11:55.584172 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.584149 5778 scope.go:122] "RemoveContainer" containerID="f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738" Nov 01 23:11:55.584448 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:11:55.584427 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738\": container with ID starting with f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738 not found: ID does not exist" containerID="f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738" Nov 01 23:11:55.584515 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.584477 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738"} err="failed to get container status \"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738\": rpc error: code = NotFound desc = could not find container \"f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738\": container with ID starting with f19d27a53f13e79fe4f3487d50bde0462c4d2ec0963a39c78c088170cd3a4738 not found: ID does not exist" Nov 01 23:11:55.595462 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.595435 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:11:55.601267 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:55.601245 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee07ad/rate-test-model-ee07ad-kserve-8476575b8f-ndvjj"] Nov 01 23:11:56.382248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.380992 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b12473d-3d7d-46fb-ae16-569af651e5da" path="/var/lib/kubelet/pods/5b12473d-3d7d-46fb-ae16-569af651e5da/volumes" Nov 01 23:11:56.382248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.381524 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 23:11:56.382248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.381770 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" podUID="affc590b-ffd9-44a5-bba1-91c84c67cccb" containerName="maas-api" containerID="cri-o://a882a3c21ecaebefaf84d6ede555c2360830b5e255fdb9606cef4ab19dc7ef25" gracePeriod=30 Nov 01 23:11:56.584568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.584510 5778 generic.go:356] "Generic (PLEG): container finished" podID="affc590b-ffd9-44a5-bba1-91c84c67cccb" containerID="a882a3c21ecaebefaf84d6ede555c2360830b5e255fdb9606cef4ab19dc7ef25" exitCode=0 Nov 01 23:11:56.584568 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.584556 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" event={"ID":"affc590b-ffd9-44a5-bba1-91c84c67cccb","Type":"ContainerDied","Data":"a882a3c21ecaebefaf84d6ede555c2360830b5e255fdb9606cef4ab19dc7ef25"} Nov 01 23:11:56.873361 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.861410 5778 scope.go:122] "RemoveContainer" containerID="30ff2db5df66807e3243cf860d60a510d55e724a741df1e4c820ddce97fbab31" Nov 01 23:11:56.907868 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.906050 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 23:11:56.926058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.926021 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 23:11:56.926300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.926268 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" containerID="cri-o://bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9" gracePeriod=30 Nov 01 23:11:56.938835 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.938780 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:11:56.942226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.940357 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" containerID="cri-o://7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd" gracePeriod=30 Nov 01 23:11:56.998999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.998698 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\") pod \"affc590b-ffd9-44a5-bba1-91c84c67cccb\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " Nov 01 23:11:56.999870 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:56.999742 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\") pod \"affc590b-ffd9-44a5-bba1-91c84c67cccb\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " Nov 01 23:11:57.000866 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.000487 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca" pod "affc590b-ffd9-44a5-bba1-91c84c67cccb" (UID: "affc590b-ffd9-44a5-bba1-91c84c67cccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:57.000866 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.000560 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\") pod \"affc590b-ffd9-44a5-bba1-91c84c67cccb\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " Nov 01 23:11:57.008000 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.007068 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs" pod "affc590b-ffd9-44a5-bba1-91c84c67cccb" (UID: "affc590b-ffd9-44a5-bba1-91c84c67cccb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:57.008297 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.008259 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq" pod "affc590b-ffd9-44a5-bba1-91c84c67cccb" (UID: "affc590b-ffd9-44a5-bba1-91c84c67cccb"). InnerVolumeSpecName "kube-api-access-fsknq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:57.013635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.012003 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\" (UniqueName: \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\") pod \"affc590b-ffd9-44a5-bba1-91c84c67cccb\" (UID: \"affc590b-ffd9-44a5-bba1-91c84c67cccb\") " Nov 01 23:11:57.013635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.012537 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/affc590b-ffd9-44a5-bba1-91c84c67cccb-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.013635 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.012768 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls" pod "affc590b-ffd9-44a5-bba1-91c84c67cccb" (UID: "affc590b-ffd9-44a5-bba1-91c84c67cccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:57.116714 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.116668 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.116714 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.116714 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/affc590b-ffd9-44a5-bba1-91c84c67cccb-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.117060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.116731 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsknq\" (UniqueName: \"kubernetes.io/projected/affc590b-ffd9-44a5-bba1-91c84c67cccb-kube-api-access-fsknq\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.592065 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.592025 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" Nov 01 23:11:57.592587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.592074 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h" event={"ID":"affc590b-ffd9-44a5-bba1-91c84c67cccb","Type":"ContainerDied","Data":"6fc8f9bb4c229c98a916d65d32819e88370ea491a4ecadd3521b53d0cb08e379"} Nov 01 23:11:57.592587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.592172 5778 scope.go:122] "RemoveContainer" containerID="a882a3c21ecaebefaf84d6ede555c2360830b5e255fdb9606cef4ab19dc7ef25" Nov 01 23:11:57.616437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.616407 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 23:11:57.620396 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:57.620372 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee07ad-74b485f9f8-ws85h"] Nov 01 23:11:58.375406 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:58.375370 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affc590b-ffd9-44a5-bba1-91c84c67cccb" path="/var/lib/kubelet/pods/affc590b-ffd9-44a5-bba1-91c84c67cccb/volumes" Nov 01 23:11:59.530907 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.530846 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerName="main" containerID="cri-o://078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a" gracePeriod=45 Nov 01 23:11:59.815050 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.814663 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:11:59.854234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854203 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\" (UniqueName: \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.854454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854253 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.854454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854314 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.854454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854416 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.854630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854543 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.854630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854621 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\") pod \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\" (UID: \"5cc550de-06fb-4a8f-817b-407ee81dcdaf\") " Nov 01 23:11:59.855070 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.854890 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:59.855210 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.855123 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:59.855430 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.855313 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:59.855521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.855448 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.855521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.855468 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.855521 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.855480 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.857309 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.857276 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:59.858391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.858371 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:59.858493 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.858413 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c" pod "5cc550de-06fb-4a8f-817b-407ee81dcdaf" (UID: "5cc550de-06fb-4a8f-817b-407ee81dcdaf"). InnerVolumeSpecName "kube-api-access-lqq2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:59.958883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.957362 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc550de-06fb-4a8f-817b-407ee81dcdaf-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.958883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.957396 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc550de-06fb-4a8f-817b-407ee81dcdaf-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.958883 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.957413 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqq2c\" (UniqueName: \"kubernetes.io/projected/5cc550de-06fb-4a8f-817b-407ee81dcdaf-kube-api-access-lqq2c\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:59.972641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.972609 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:11:59.974139 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:11:59.973896 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" containerID="cri-o://6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08" gracePeriod=30 Nov 01 23:12:00.595700 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.595657 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:00.596234 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.595739 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:00.613679 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.613649 5778 generic.go:356] "Generic (PLEG): container finished" podID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerID="078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a" exitCode=0 Nov 01 23:12:00.613892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.613695 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" Nov 01 23:12:00.613892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.613729 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" event={"ID":"5cc550de-06fb-4a8f-817b-407ee81dcdaf","Type":"ContainerDied","Data":"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a"} Nov 01 23:12:00.613892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.613772 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" event={"ID":"5cc550de-06fb-4a8f-817b-407ee81dcdaf","Type":"ContainerDied","Data":"1b1aa037d057b8f2d6b912f1b0ed41645bdca2aaf95c1701678631e178f7032c"} Nov 01 23:12:00.613892 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.613788 5778 scope.go:122] "RemoveContainer" containerID="078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a" Nov 01 23:12:00.625690 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.625614 5778 scope.go:122] "RemoveContainer" containerID="078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a" Nov 01 23:12:00.626079 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:00.626053 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a\": container with ID starting with 078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a not found: ID does not exist" containerID="078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a" Nov 01 23:12:00.626153 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.626093 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a"} err="failed to get container status \"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a\": rpc error: code = NotFound desc = could not find container \"078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a\": container with ID starting with 078996cd537e0b95624d70c9d13e365188ec381292bd0baa8ebec82ff4f8ee4a not found: ID does not exist" Nov 01 23:12:00.633678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.633619 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:12:00.637526 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.637500 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956"] Nov 01 23:12:00.720245 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:00.720216 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dc635e/rate-test-model-dc635e-kserve-64586cc89c-ww956" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:12:01.299926 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:01.299889 5778 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:01.300135 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:01.299953 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:02.323896 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.322464 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:12:02.326891 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.326872 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 23:12:02.377311 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.377281 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" path="/var/lib/kubelet/pods/5cc550de-06fb-4a8f-817b-407ee81dcdaf/volumes" Nov 01 23:12:02.387083 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387060 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387114 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387143 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387171 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.387259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387200 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387259 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387230 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.387494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387296 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387325 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387367 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.387494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387392 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.387494 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387401 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.387726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387513 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.387726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387555 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.387726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387603 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.387726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.387630 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:02.388391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.388327 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.388811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.388785 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.388989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.388817 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.388989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.388914 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.388989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.388959 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.389190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389000 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.389190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389055 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\") pod \"f1759788-2268-4a5d-9c80-ec6a2b39d528\" (UID: \"f1759788-2268-4a5d-9c80-ec6a2b39d528\") " Nov 01 23:12:02.389190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389083 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\") pod \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\" (UID: \"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1\") " Nov 01 23:12:02.389190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389096 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.389418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389345 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.389603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389351 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.389603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389574 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.389603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389580 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389617 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389636 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389676 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389690 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389703 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389718 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.389889 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.389794 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:02.390708 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.390683 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.390831 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.390736 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.391540 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.391509 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.392963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.392511 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:02.392963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.392634 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.392963 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.392836 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "kube-api-access-5jlbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.393216 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.393104 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf" pod "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" (UID: "e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1"). InnerVolumeSpecName "kube-api-access-b85vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.393377 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.393358 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo" pod "f1759788-2268-4a5d-9c80-ec6a2b39d528" (UID: "f1759788-2268-4a5d-9c80-ec6a2b39d528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:02.491405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491322 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491358 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85vf\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-kube-api-access-b85vf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491375 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491389 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491402 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1759788-2268-4a5d-9c80-ec6a2b39d528-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491416 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491429 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491440 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491452 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491464 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1759788-2268-4a5d-9c80-ec6a2b39d528-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.491726 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.491478 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jlbj\" (UniqueName: \"kubernetes.io/projected/f1759788-2268-4a5d-9c80-ec6a2b39d528-kube-api-access-5jlbj\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.633481 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.633443 5778 generic.go:356] "Generic (PLEG): container finished" podID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerID="bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9" exitCode=0 Nov 01 23:12:02.633693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.633507 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" Nov 01 23:12:02.633693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.633532 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerDied","Data":"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9"} Nov 01 23:12:02.633693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.633597 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf" event={"ID":"e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1","Type":"ContainerDied","Data":"bc58af184fec17a9049874692b2d3ee3a987d3da1afd07e05c35f116afbcbaa5"} Nov 01 23:12:02.633693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.633629 5778 scope.go:122] "RemoveContainer" containerID="bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9" Nov 01 23:12:02.636161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.636133 5778 generic.go:356] "Generic (PLEG): container finished" podID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerID="7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd" exitCode=0 Nov 01 23:12:02.636322 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.636214 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" Nov 01 23:12:02.636322 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.636235 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" event={"ID":"f1759788-2268-4a5d-9c80-ec6a2b39d528","Type":"ContainerDied","Data":"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd"} Nov 01 23:12:02.636322 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.636280 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2" event={"ID":"f1759788-2268-4a5d-9c80-ec6a2b39d528","Type":"ContainerDied","Data":"c366199d50fcc5f89888d07bf24ec4ca86fd5dc18ba5cf754c2bb83f127e766b"} Nov 01 23:12:02.662730 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.662706 5778 scope.go:122] "RemoveContainer" containerID="87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a" Nov 01 23:12:02.673947 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.673920 5778 scope.go:122] "RemoveContainer" containerID="bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9" Nov 01 23:12:02.674311 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:02.674282 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9\": container with ID starting with bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9 not found: ID does not exist" containerID="bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9" Nov 01 23:12:02.674417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.674321 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9"} err="failed to get container status \"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9\": rpc error: code = NotFound desc = could not find container \"bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9\": container with ID starting with bdbee0e979f791612475b96b4d81eaa93068579003a6a13bb497f0c57baf43e9 not found: ID does not exist" Nov 01 23:12:02.674417 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.674340 5778 scope.go:122] "RemoveContainer" containerID="87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a" Nov 01 23:12:02.674666 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:02.674636 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a\": container with ID starting with 87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a not found: ID does not exist" containerID="87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a" Nov 01 23:12:02.674733 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.674673 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a"} err="failed to get container status \"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a\": rpc error: code = NotFound desc = could not find container \"87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a\": container with ID starting with 87856368dbcd441a8bf39ee664f9d298f4b7a83e0aeda5b433884f5dab97001a not found: ID does not exist" Nov 01 23:12:02.674733 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.674692 5778 scope.go:122] "RemoveContainer" containerID="7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd" Nov 01 23:12:02.683458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.683432 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 23:12:02.687019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.686994 5778 scope.go:122] "RemoveContainer" containerID="7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd" Nov 01 23:12:02.689689 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.689666 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-2z8tf"] Nov 01 23:12:02.689972 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:02.689950 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd\": container with ID starting with 7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd not found: ID does not exist" containerID="7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd" Nov 01 23:12:02.690060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.689982 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd"} err="failed to get container status \"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd\": rpc error: code = NotFound desc = could not find container \"7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd\": container with ID starting with 7b3f374ca12f9c934977121dad0d0a68cb79d5b38bd602136a730b2211cd73cd not found: ID does not exist" Nov 01 23:12:02.703359 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.703329 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:12:02.710437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:02.710407 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sktn2"] Nov 01 23:12:04.374396 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:04.374360 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" path="/var/lib/kubelet/pods/e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1/volumes" Nov 01 23:12:04.374881 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:04.374846 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" path="/var/lib/kubelet/pods/f1759788-2268-4a5d-9c80-ec6a2b39d528/volumes" Nov 01 23:12:05.292520 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.292489 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:12:05.435893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.435842 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436005 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436037 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436072 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436100 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436183 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436222 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436257 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436372 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436319 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\") pod \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\" (UID: \"e3ffb239-7a92-4b4a-bd57-30c90c3d2942\") " Nov 01 23:12:05.436836 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436610 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.436836 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436775 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.436992 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436890 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.436992 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436904 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:05.437117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.436998 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.437117 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.437071 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.438780 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.438754 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:05.439050 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.439015 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "kube-api-access-2n92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:05.439050 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.439037 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:05.439388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.439368 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy" pod "e3ffb239-7a92-4b4a-bd57-30c90c3d2942" (UID: "e3ffb239-7a92-4b4a-bd57-30c90c3d2942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.499003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.498964 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 23:12:05.532385 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.532287 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 23:12:05.538156 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538128 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n92k\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-kube-api-access-2n92k\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538156 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538156 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538166 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538176 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538186 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538195 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538204 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.538333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.538212 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffb239-7a92-4b4a-bd57-30c90c3d2942-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.663862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.663817 5778 generic.go:356] "Generic (PLEG): container finished" podID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerID="6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08" exitCode=0 Nov 01 23:12:05.664073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.663898 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" event={"ID":"e3ffb239-7a92-4b4a-bd57-30c90c3d2942","Type":"ContainerDied","Data":"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08"} Nov 01 23:12:05.664073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.663950 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" event={"ID":"e3ffb239-7a92-4b4a-bd57-30c90c3d2942","Type":"ContainerDied","Data":"674cc16d8627588155da98f279a6fdd8a2b9eae82affbd5663407ecf7af0a898"} Nov 01 23:12:05.664073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.663978 5778 scope.go:122] "RemoveContainer" containerID="6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08" Nov 01 23:12:05.664282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.664122 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc" Nov 01 23:12:05.691136 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.691044 5778 scope.go:122] "RemoveContainer" containerID="6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08" Nov 01 23:12:05.691450 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:05.691420 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08\": container with ID starting with 6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08 not found: ID does not exist" containerID="6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08" Nov 01 23:12:05.691564 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.691462 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08"} err="failed to get container status \"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08\": rpc error: code = NotFound desc = could not find container \"6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08\": container with ID starting with 6fb2080de0792fa7e065ce33312e0216b80fedef9f8ef36991b1bcd1d4a08b08 not found: ID does not exist" Nov 01 23:12:05.707335 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.707292 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:12:05.753874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.752914 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6hgrc"] Nov 01 23:12:05.754864 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.754823 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 23:12:05.755129 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.755100 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" containerID="cri-o://bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08" gracePeriod=30 Nov 01 23:12:05.988997 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.988963 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 23:12:05.989327 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:05.989272 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" containerID="cri-o://8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23" gracePeriod=30 Nov 01 23:12:06.376584 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:06.376506 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" path="/var/lib/kubelet/pods/e3ffb239-7a92-4b4a-bd57-30c90c3d2942/volumes" Nov 01 23:12:11.117571 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.117534 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244358 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244420 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244477 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244508 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244552 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244594 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244695 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244731 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.244805 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\") pod \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\" (UID: \"d98c93a2-a15f-4e57-bd94-e5b685a879d4\") " Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.245068 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.245277 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.245874 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.245547 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.246674 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.246018 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.246674 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.246382 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.246674 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.246402 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.250894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.248595 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "kube-api-access-whc48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.250894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.249299 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.250894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.249729 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.250894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.250672 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo" pod "d98c93a2-a15f-4e57-bd94-e5b685a879d4" (UID: "d98c93a2-a15f-4e57-bd94-e5b685a879d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:11.283517 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.283486 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 23:12:11.345742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.345647 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.345742 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.345724 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.345779 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.345930 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.345983 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346017 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346019 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346056 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346132 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346162 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\") pod \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\" (UID: \"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd\") " Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346257 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346318 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346819 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346844 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346897 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346914 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346930 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whc48\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-kube-api-access-whc48\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346945 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346959 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346974 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.346988 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.347002 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c93a2-a15f-4e57-bd94-e5b685a879d4-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.347017 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c93a2-a15f-4e57-bd94-e5b685a879d4-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.347383 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.347735 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.347523 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.349455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.348951 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.349596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.349565 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:11.349596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.349585 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "kube-api-access-wqpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.350266 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.350250 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy" pod "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" (UID: "c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.447718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447687 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.447718 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447719 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.447983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447733 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.447983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447747 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.447983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447759 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.447983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.447771 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqpxg\" (UniqueName: \"kubernetes.io/projected/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd-kube-api-access-wqpxg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.728058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.728017 5778 generic.go:356] "Generic (PLEG): container finished" podID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerID="bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08" exitCode=0 Nov 01 23:12:11.728278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.728086 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" Nov 01 23:12:11.728278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.728192 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerDied","Data":"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08"} Nov 01 23:12:11.728278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.728235 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn" event={"ID":"d98c93a2-a15f-4e57-bd94-e5b685a879d4","Type":"ContainerDied","Data":"99d8c55a8cdc39a7d09fa309367c8dd0d96c39821eef739994e97f6c8585e022"} Nov 01 23:12:11.728278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.728261 5778 scope.go:122] "RemoveContainer" containerID="bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08" Nov 01 23:12:11.734519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.730607 5778 generic.go:356] "Generic (PLEG): container finished" podID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerID="8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23" exitCode=0 Nov 01 23:12:11.734519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.730646 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" event={"ID":"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd","Type":"ContainerDied","Data":"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23"} Nov 01 23:12:11.734519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.730651 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" Nov 01 23:12:11.734519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.730689 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz" event={"ID":"c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd","Type":"ContainerDied","Data":"e94eab422ba5395c43453bb3db274e21d2bfa7c2cae6a86e0d7133ff585accb7"} Nov 01 23:12:11.743542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.743518 5778 scope.go:122] "RemoveContainer" containerID="e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3" Nov 01 23:12:11.756029 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.756001 5778 scope.go:122] "RemoveContainer" containerID="bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08" Nov 01 23:12:11.756359 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:11.756331 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08\": container with ID starting with bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08 not found: ID does not exist" containerID="bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08" Nov 01 23:12:11.756476 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.756366 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08"} err="failed to get container status \"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08\": rpc error: code = NotFound desc = could not find container \"bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08\": container with ID starting with bc7149dee68819d77d579dbe50281f9ac961d2def544cb3ec5ca6458e7e23e08 not found: ID does not exist" Nov 01 23:12:11.756476 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.756384 5778 scope.go:122] "RemoveContainer" containerID="e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3" Nov 01 23:12:11.756682 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:11.756653 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3\": container with ID starting with e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3 not found: ID does not exist" containerID="e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3" Nov 01 23:12:11.756745 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.756687 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3"} err="failed to get container status \"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3\": rpc error: code = NotFound desc = could not find container \"e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3\": container with ID starting with e3c1d55bd7fdfb2122e89cb80422511a0cb836641a4ff1a1c77d89dfba420cb3 not found: ID does not exist" Nov 01 23:12:11.756745 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.756703 5778 scope.go:122] "RemoveContainer" containerID="8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23" Nov 01 23:12:11.759166 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.759116 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 23:12:11.761683 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.761643 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-xv7pn"] Nov 01 23:12:11.768887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.768867 5778 scope.go:122] "RemoveContainer" containerID="8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23" Nov 01 23:12:11.769199 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:11.769177 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23\": container with ID starting with 8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23 not found: ID does not exist" containerID="8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23" Nov 01 23:12:11.769301 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.769207 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23"} err="failed to get container status \"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23\": rpc error: code = NotFound desc = could not find container \"8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23\": container with ID starting with 8f3d7418a7f60806e5ecd16cd02244d1972c8aaeea934a2348bcfa5f9d52ab23 not found: ID does not exist" Nov 01 23:12:11.772827 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.772806 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 23:12:11.779895 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:11.779871 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-n2mwz"] Nov 01 23:12:12.374260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:12.374228 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" path="/var/lib/kubelet/pods/c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd/volumes" Nov 01 23:12:12.374693 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:12.374679 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" path="/var/lib/kubelet/pods/d98c93a2-a15f-4e57-bd94-e5b685a879d4/volumes" Nov 01 23:12:18.773250 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:18.773210 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:12:20.520812 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.520754 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" containerID="cri-o://2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc" gracePeriod=45 Nov 01 23:12:20.553907 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.553872 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" containerID="cri-o://445481322dd104dc5497902b4b20f6d4f93553de1a783ef35ccc14e8c5ce5a8b" gracePeriod=45 Nov 01 23:12:20.819702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.819674 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 23:12:20.820227 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.820200 5778 generic.go:356] "Generic (PLEG): container finished" podID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerID="445481322dd104dc5497902b4b20f6d4f93553de1a783ef35ccc14e8c5ce5a8b" exitCode=0 Nov 01 23:12:20.820366 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.820290 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" event={"ID":"81b50e76-eec9-41fa-8a9d-c708e67ac7a2","Type":"ContainerDied","Data":"445481322dd104dc5497902b4b20f6d4f93553de1a783ef35ccc14e8c5ce5a8b"} Nov 01 23:12:20.822951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.821822 5778 generic.go:356] "Generic (PLEG): container finished" podID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerID="2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc" exitCode=0 Nov 01 23:12:20.822951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.821885 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" Nov 01 23:12:20.822951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.821881 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" event={"ID":"c560e2a1-64d9-4270-ba56-bd2217f2a687","Type":"ContainerDied","Data":"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc"} Nov 01 23:12:20.822951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.822017 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j" event={"ID":"c560e2a1-64d9-4270-ba56-bd2217f2a687","Type":"ContainerDied","Data":"deee3a1b06bc6db539c53332a80ba36ff7cf912304d5622504267fa350f624b0"} Nov 01 23:12:20.822951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.822045 5778 scope.go:122] "RemoveContainer" containerID="2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc" Nov 01 23:12:20.849103 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.849050 5778 scope.go:122] "RemoveContainer" containerID="2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc" Nov 01 23:12:20.849518 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:20.849486 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc\": container with ID starting with 2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc not found: ID does not exist" containerID="2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc" Nov 01 23:12:20.849626 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.849531 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc"} err="failed to get container status \"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc\": rpc error: code = NotFound desc = could not find container \"2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc\": container with ID starting with 2ce7aa9f6570fb7225ca00f81ff88404fa1a53107434045ee30a23e509e6a8bc not found: ID does not exist" Nov 01 23:12:20.863658 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.863632 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.863897 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.863840 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\" (UniqueName: \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.864128 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864111 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.863937 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864279 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864355 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864387 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\" (UniqueName: \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\") pod \"c560e2a1-64d9-4270-ba56-bd2217f2a687\" (UID: \"c560e2a1-64d9-4270-ba56-bd2217f2a687\") " Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864501 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.864705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.864666 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.865664 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.865633 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.865664 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.865663 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.865840 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.865700 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.867525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.867486 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.867525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.867496 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "kube-api-access-hnndh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:20.869028 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.868995 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs" pod "c560e2a1-64d9-4270-ba56-bd2217f2a687" (UID: "c560e2a1-64d9-4270-ba56-bd2217f2a687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:20.882523 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.882499 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 23:12:20.966465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966431 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.966652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966499 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.966652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966532 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\" (UniqueName: \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.966652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966590 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.966652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966627 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.966957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.966728 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\") pod \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\" (UID: \"81b50e76-eec9-41fa-8a9d-c708e67ac7a2\") " Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967191 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c560e2a1-64d9-4270-ba56-bd2217f2a687-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967219 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c560e2a1-64d9-4270-ba56-bd2217f2a687-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967235 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnndh\" (UniqueName: \"kubernetes.io/projected/c560e2a1-64d9-4270-ba56-bd2217f2a687-kube-api-access-hnndh\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967304 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967369 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.967576 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.967493 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.969348 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.969320 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.969965 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.969940 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:20.970289 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:20.970261 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7" pod "81b50e76-eec9-41fa-8a9d-c708e67ac7a2" (UID: "81b50e76-eec9-41fa-8a9d-c708e67ac7a2"). InnerVolumeSpecName "kube-api-access-6k8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071232 5778 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-model-cache\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071302 5778 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-dshm\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071313 5778 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tls-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071322 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8j7\" (UniqueName: \"kubernetes.io/projected/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-kube-api-access-6k8j7\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071333 5778 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-tmp-dir\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.071388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.071345 5778 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b50e76-eec9-41fa-8a9d-c708e67ac7a2-home\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.146729 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.146691 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 23:12:21.151524 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.151494 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-w3-6ac182-kserve-579d65c59c-5f76j"] Nov 01 23:12:21.831367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.831275 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" Nov 01 23:12:21.831367 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.831313 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr" event={"ID":"81b50e76-eec9-41fa-8a9d-c708e67ac7a2","Type":"ContainerDied","Data":"e7d5feb741af10a3825de23b9fa83f1083034faef1a07384a056fb03a7d127fd"} Nov 01 23:12:21.831955 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.831367 5778 scope.go:122] "RemoveContainer" containerID="445481322dd104dc5497902b4b20f6d4f93553de1a783ef35ccc14e8c5ce5a8b" Nov 01 23:12:21.862054 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.862025 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 23:12:21.869720 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:21.869677 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/simulator-w3-6ac182-kserve-6d8f9dc6c9-j7xnr"] Nov 01 23:12:22.373527 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:22.373495 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" path="/var/lib/kubelet/pods/81b50e76-eec9-41fa-8a9d-c708e67ac7a2/volumes" Nov 01 23:12:22.373967 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:22.373952 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" path="/var/lib/kubelet/pods/c560e2a1-64d9-4270-ba56-bd2217f2a687/volumes" Nov 01 23:12:26.057717 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.057679 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 23:12:26.058257 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.058091 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" podUID="5efec5dc-2357-4a61-9fe5-3e73b516be01" containerName="maas-api" containerID="cri-o://ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c" gracePeriod=30 Nov 01 23:12:26.351478 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.351451 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 23:12:26.425546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.425516 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\") pod \"5efec5dc-2357-4a61-9fe5-3e73b516be01\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " Nov 01 23:12:26.425739 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.425584 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\") pod \"5efec5dc-2357-4a61-9fe5-3e73b516be01\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " Nov 01 23:12:26.425739 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.425627 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\") pod \"5efec5dc-2357-4a61-9fe5-3e73b516be01\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " Nov 01 23:12:26.425739 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.425671 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\") pod \"5efec5dc-2357-4a61-9fe5-3e73b516be01\" (UID: \"5efec5dc-2357-4a61-9fe5-3e73b516be01\") " Nov 01 23:12:26.426615 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.426586 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca" pod "5efec5dc-2357-4a61-9fe5-3e73b516be01" (UID: "5efec5dc-2357-4a61-9fe5-3e73b516be01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:26.428505 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.428464 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz" pod "5efec5dc-2357-4a61-9fe5-3e73b516be01" (UID: "5efec5dc-2357-4a61-9fe5-3e73b516be01"). InnerVolumeSpecName "kube-api-access-8zchz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:26.428643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.428565 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls" pod "5efec5dc-2357-4a61-9fe5-3e73b516be01" (UID: "5efec5dc-2357-4a61-9fe5-3e73b516be01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:26.428643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.428592 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs" pod "5efec5dc-2357-4a61-9fe5-3e73b516be01" (UID: "5efec5dc-2357-4a61-9fe5-3e73b516be01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:26.526922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.526883 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:26.526922 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.526922 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5efec5dc-2357-4a61-9fe5-3e73b516be01-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:26.527230 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.526937 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efec5dc-2357-4a61-9fe5-3e73b516be01-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:26.527230 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.526952 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/5efec5dc-2357-4a61-9fe5-3e73b516be01-kube-api-access-8zchz\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:26.650970 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.650799 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:12:26.651502 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.651469 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" containerID="cri-o://effc8aa9915de506545562b7e24e36c1b83fdf7007d5a1bb5a9da41cfcfec5d5" gracePeriod=30 Nov 01 23:12:26.864356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.864323 5778 generic.go:356] "Generic (PLEG): container finished" podID="5efec5dc-2357-4a61-9fe5-3e73b516be01" containerID="ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c" exitCode=0 Nov 01 23:12:26.864509 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.864365 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" Nov 01 23:12:26.864509 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.864399 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" event={"ID":"5efec5dc-2357-4a61-9fe5-3e73b516be01","Type":"ContainerDied","Data":"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c"} Nov 01 23:12:26.864509 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.864453 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87" event={"ID":"5efec5dc-2357-4a61-9fe5-3e73b516be01","Type":"ContainerDied","Data":"aa9456d18ba383e5169dcb7ed30dde0ed43f5ca45b1b4a3633912af01af9a894"} Nov 01 23:12:26.864509 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.864476 5778 scope.go:122] "RemoveContainer" containerID="ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c" Nov 01 23:12:26.877197 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.877178 5778 scope.go:122] "RemoveContainer" containerID="ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c" Nov 01 23:12:26.877473 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:26.877451 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c\": container with ID starting with ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c not found: ID does not exist" containerID="ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c" Nov 01 23:12:26.877547 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.877481 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c"} err="failed to get container status \"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c\": rpc error: code = NotFound desc = could not find container \"ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c\": container with ID starting with ef8110edd22854a5c0d54b59d1f928ce701898adc7b650b75e381b190b7f7b9c not found: ID does not exist" Nov 01 23:12:26.889151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.889128 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 23:12:26.893079 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:26.893055 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e26733-6577bf869d-ctr87"] Nov 01 23:12:27.214073 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.214041 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 23:12:27.215448 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.214283 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" podUID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" containerName="maas-api" containerID="cri-o://621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f" gracePeriod=30 Nov 01 23:12:27.537356 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.537332 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 23:12:27.638555 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.638517 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\" (UniqueName: \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\") pod \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " Nov 01 23:12:27.638774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.638573 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\") pod \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " Nov 01 23:12:27.638774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.638621 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\") pod \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " Nov 01 23:12:27.638924 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.638770 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\") pod \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\" (UID: \"2c149a45-0908-430e-9f23-c3dabb1b9ea7\") " Nov 01 23:12:27.639074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.639049 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca" pod "2c149a45-0908-430e-9f23-c3dabb1b9ea7" (UID: "2c149a45-0908-430e-9f23-c3dabb1b9ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:27.639237 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.639213 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c149a45-0908-430e-9f23-c3dabb1b9ea7-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:27.641119 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.641096 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs" pod "2c149a45-0908-430e-9f23-c3dabb1b9ea7" (UID: "2c149a45-0908-430e-9f23-c3dabb1b9ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:27.641233 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.641165 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2" pod "2c149a45-0908-430e-9f23-c3dabb1b9ea7" (UID: "2c149a45-0908-430e-9f23-c3dabb1b9ea7"). InnerVolumeSpecName "kube-api-access-tb5j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:27.641804 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.641788 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls" pod "2c149a45-0908-430e-9f23-c3dabb1b9ea7" (UID: "2c149a45-0908-430e-9f23-c3dabb1b9ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:27.740383 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.740351 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:27.740383 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.740384 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5j2\" (UniqueName: \"kubernetes.io/projected/2c149a45-0908-430e-9f23-c3dabb1b9ea7-kube-api-access-tb5j2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:27.740541 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.740399 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c149a45-0908-430e-9f23-c3dabb1b9ea7-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:27.872184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.872103 5778 generic.go:356] "Generic (PLEG): container finished" podID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" containerID="621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f" exitCode=0 Nov 01 23:12:27.872184 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.872155 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" Nov 01 23:12:27.872409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.872186 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" event={"ID":"2c149a45-0908-430e-9f23-c3dabb1b9ea7","Type":"ContainerDied","Data":"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f"} Nov 01 23:12:27.872409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.872244 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv" event={"ID":"2c149a45-0908-430e-9f23-c3dabb1b9ea7","Type":"ContainerDied","Data":"ca48b8de6fbff572f14ed204fe77c63fbe16086a2d96cfd7aab450bfce01235e"} Nov 01 23:12:27.872409 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.872270 5778 scope.go:122] "RemoveContainer" containerID="621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f" Nov 01 23:12:27.882929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.882913 5778 scope.go:122] "RemoveContainer" containerID="621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f" Nov 01 23:12:27.883217 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:27.883197 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f\": container with ID starting with 621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f not found: ID does not exist" containerID="621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f" Nov 01 23:12:27.883287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.883224 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f"} err="failed to get container status \"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f\": rpc error: code = NotFound desc = could not find container \"621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f\": container with ID starting with 621adbac11031fb9590666dc833ffe3c53403e673a76cd7e475265c8a70dc47f not found: ID does not exist" Nov 01 23:12:27.894386 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.894368 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 23:12:27.902612 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:27.902591 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6ac182-9c65f8677-mcrnv"] Nov 01 23:12:28.373163 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:28.373131 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" path="/var/lib/kubelet/pods/2c149a45-0908-430e-9f23-c3dabb1b9ea7/volumes" Nov 01 23:12:28.373567 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:28.373538 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efec5dc-2357-4a61-9fe5-3e73b516be01" path="/var/lib/kubelet/pods/5efec5dc-2357-4a61-9fe5-3e73b516be01/volumes" Nov 01 23:12:31.913199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:31.913154 5778 generic.go:356] "Generic (PLEG): container finished" podID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerID="effc8aa9915de506545562b7e24e36c1b83fdf7007d5a1bb5a9da41cfcfec5d5" exitCode=0 Nov 01 23:12:31.913737 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:31.913332 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" event={"ID":"8fdd665f-021d-4fa4-a753-d0051b31c6d8","Type":"ContainerDied","Data":"effc8aa9915de506545562b7e24e36c1b83fdf7007d5a1bb5a9da41cfcfec5d5"} Nov 01 23:12:31.948632 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:31.948560 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:12:32.084741 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084699 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.084999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084767 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.084999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084807 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.084999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084839 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.084999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084928 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.084999 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.084978 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.085300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085004 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.085300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085041 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.085300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085069 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\") pod \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\" (UID: \"8fdd665f-021d-4fa4-a753-d0051b31c6d8\") " Nov 01 23:12:32.085300 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085106 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.085458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085432 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.086046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.085999 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.086287 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.086249 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:32.086498 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.086467 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.087418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.087387 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.093425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.093385 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.093547 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.093526 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "kube-api-access-6mr5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.093788 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.093770 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:32.095290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.095268 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy" pod "8fdd665f-021d-4fa4-a753-d0051b31c6d8" (UID: "8fdd665f-021d-4fa4-a753-d0051b31c6d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.186652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186610 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186652 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186669 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186682 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186696 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186710 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdd665f-021d-4fa4-a753-d0051b31c6d8-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186724 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdd665f-021d-4fa4-a753-d0051b31c6d8-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.186956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.186738 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mr5j\" (UniqueName: \"kubernetes.io/projected/8fdd665f-021d-4fa4-a753-d0051b31c6d8-kube-api-access-6mr5j\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.921662 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.921612 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" event={"ID":"8fdd665f-021d-4fa4-a753-d0051b31c6d8","Type":"ContainerDied","Data":"426b16f5d795ec8d28990420ddc17b0473c437294e511f4cf16eb49525d46aa1"} Nov 01 23:12:32.922190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.921685 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf" Nov 01 23:12:32.922190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.921688 5778 scope.go:122] "RemoveContainer" containerID="effc8aa9915de506545562b7e24e36c1b83fdf7007d5a1bb5a9da41cfcfec5d5" Nov 01 23:12:32.943542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.943501 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:12:32.951752 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:32.951729 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-ddtdf"] Nov 01 23:12:34.374312 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:34.374275 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" path="/var/lib/kubelet/pods/8fdd665f-021d-4fa4-a753-d0051b31c6d8/volumes" Nov 01 23:12:34.569003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:34.568959 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 23:12:34.569254 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:34.569222 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" containerID="cri-o://146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752" gracePeriod=30 Nov 01 23:12:35.712912 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:35.712875 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 23:12:35.944841 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:35.944788 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" containerID="cri-o://42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318" gracePeriod=30 Nov 01 23:12:36.364038 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:36.363996 5778 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:36.364262 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:36.364071 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:39.856761 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.856686 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 23:12:39.970546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970516 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970597 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970625 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970665 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970708 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970764 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.970814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970791 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.971195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970819 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.971195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970835 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.971195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.970885 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\") pod \"cc37c643-dc48-4601-b18a-acd7262df3f3\" (UID: \"cc37c643-dc48-4601-b18a-acd7262df3f3\") " Nov 01 23:12:39.971195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971096 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.971195 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971151 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971262 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971336 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971360 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971360 5778 generic.go:356] "Generic (PLEG): container finished" podID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerID="146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752" exitCode=0 Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971399 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" event={"ID":"cc37c643-dc48-4601-b18a-acd7262df3f3","Type":"ContainerDied","Data":"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752"} Nov 01 23:12:39.971437 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971370 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.971778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971443 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" Nov 01 23:12:39.971778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971466 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp" event={"ID":"cc37c643-dc48-4601-b18a-acd7262df3f3","Type":"ContainerDied","Data":"d176122ba6fb76bae9c877b3c401bba2aeff80fb23feee4a41b189767c420c24"} Nov 01 23:12:39.971778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971499 5778 scope.go:122] "RemoveContainer" containerID="146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752" Nov 01 23:12:39.971998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.971794 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.973650 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.973620 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.973774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.973712 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "kube-api-access-lncz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:39.973882 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.973826 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:39.974180 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.974157 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token" pod "cc37c643-dc48-4601-b18a-acd7262df3f3" (UID: "cc37c643-dc48-4601-b18a-acd7262df3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:39.997846 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.997822 5778 scope.go:122] "RemoveContainer" containerID="146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752" Nov 01 23:12:39.998249 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:39.998227 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752\": container with ID starting with 146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752 not found: ID does not exist" containerID="146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752" Nov 01 23:12:39.998345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:39.998264 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752"} err="failed to get container status \"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752\": rpc error: code = NotFound desc = could not find container \"146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752\": container with ID starting with 146ae16a399293724b0cc9d82c873c3472f315624530eeaa4abd34269753e752 not found: ID does not exist" Nov 01 23:12:40.072465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072430 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lncz2\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-kube-api-access-lncz2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.072465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072458 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc37c643-dc48-4601-b18a-acd7262df3f3-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.072465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072467 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.072465 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072476 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc37c643-dc48-4601-b18a-acd7262df3f3-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.072754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072485 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.072754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.072493 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc37c643-dc48-4601-b18a-acd7262df3f3-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.318758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.318719 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 23:12:40.323466 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.323438 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-z7csp"] Nov 01 23:12:40.377389 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:40.375833 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" path="/var/lib/kubelet/pods/cc37c643-dc48-4601-b18a-acd7262df3f3/volumes" Nov 01 23:12:41.211814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.211775 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 23:12:41.286725 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286649 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.286725 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286697 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.286725 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286724 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286757 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286797 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286816 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286884 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286954 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286961 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.287046 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.286976 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\") pod \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\" (UID: \"db312c02-f1b1-4eb0-809a-3bf6bdef2776\") " Nov 01 23:12:41.287429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287056 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.287429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287198 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.287429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287321 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.287587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287471 5778 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-data\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.287587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287493 5778 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istiod-ca-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.287587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287509 5778 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-credential-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.287587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287522 5778 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-socket\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.287587 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.287578 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.289761 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.289713 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:41.289905 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.289761 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.290226 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.290172 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.290338 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.290248 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg" pod "db312c02-f1b1-4eb0-809a-3bf6bdef2776" (UID: "db312c02-f1b1-4eb0-809a-3bf6bdef2776"). InnerVolumeSpecName "kube-api-access-8c4mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.388914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.388874 5778 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-podinfo\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.388914 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.388917 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4mg\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-kube-api-access-8c4mg\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.389159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.388932 5778 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-envoy\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.389159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.388945 5778 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db312c02-f1b1-4eb0-809a-3bf6bdef2776-workload-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.389159 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.388958 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db312c02-f1b1-4eb0-809a-3bf6bdef2776-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.987376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.987342 5778 generic.go:356] "Generic (PLEG): container finished" podID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerID="42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318" exitCode=0 Nov 01 23:12:41.987562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.987403 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" Nov 01 23:12:41.987562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.987414 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" event={"ID":"db312c02-f1b1-4eb0-809a-3bf6bdef2776","Type":"ContainerDied","Data":"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318"} Nov 01 23:12:41.987562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.987473 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8" event={"ID":"db312c02-f1b1-4eb0-809a-3bf6bdef2776","Type":"ContainerDied","Data":"211619961b86558d9bf7b04694b4eb7a8fc317dc80e24949f81c7d4fb6b23f79"} Nov 01 23:12:41.987562 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.987494 5778 scope.go:122] "RemoveContainer" containerID="42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318" Nov 01 23:12:41.998911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.998893 5778 scope.go:122] "RemoveContainer" containerID="42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318" Nov 01 23:12:41.999200 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:41.999178 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318\": container with ID starting with 42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318 not found: ID does not exist" containerID="42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318" Nov 01 23:12:41.999238 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:41.999210 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318"} err="failed to get container status \"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318\": rpc error: code = NotFound desc = could not find container \"42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318\": container with ID starting with 42976ce1bf4da57f81d374202644d9c623f6bdc63c7915c62bd0bac2b30d5318 not found: ID does not exist" Nov 01 23:12:42.011630 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:42.011597 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 23:12:42.018413 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:42.018384 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-497r8"] Nov 01 23:12:42.373788 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:12:42.373704 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" path="/var/lib/kubelet/pods/db312c02-f1b1-4eb0-809a-3bf6bdef2776/volumes" Nov 01 23:12:48.773365 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:12:48.773323 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:13:18.772522 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:13:18.772486 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:13:20.486933 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.486894 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9"] Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487454 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487474 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487499 5778 cpu_manager.go:414] "containerMap: removing container" podUID="1ddf4019-13be-4213-937a-78ef85850d51" containerName="maas-api" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487507 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddf4019-13be-4213-937a-78ef85850d51" containerName="maas-api" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487517 5778 cpu_manager.go:414] "containerMap: removing container" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487525 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487539 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487546 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487559 5778 cpu_manager.go:414] "containerMap: removing container" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487567 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487579 5778 cpu_manager.go:414] "containerMap: removing container" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487587 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487598 5778 cpu_manager.go:414] "containerMap: removing container" podUID="affc590b-ffd9-44a5-bba1-91c84c67cccb" containerName="maas-api" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487605 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affc590b-ffd9-44a5-bba1-91c84c67cccb" containerName="maas-api" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487615 5778 cpu_manager.go:414] "containerMap: removing container" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487622 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487631 5778 cpu_manager.go:414] "containerMap: removing container" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" Nov 01 23:13:20.487634 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487637 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487646 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487652 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487665 5778 cpu_manager.go:414] "containerMap: removing container" podUID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487672 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487683 5778 cpu_manager.go:414] "containerMap: removing container" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487690 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487698 5778 cpu_manager.go:414] "containerMap: removing container" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487703 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487713 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487717 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487723 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d448a01a-e368-4968-8912-9d8b1cb19f8f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487727 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d448a01a-e368-4968-8912-9d8b1cb19f8f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487733 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487737 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487746 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487750 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487755 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8fe315b0-6760-49d9-b333-5531f0c66cad" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487759 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe315b0-6760-49d9-b333-5531f0c66cad" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487766 5778 cpu_manager.go:414] "containerMap: removing container" podUID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487770 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487776 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487781 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487788 5778 cpu_manager.go:414] "containerMap: removing container" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487793 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487800 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487805 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487810 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487814 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487819 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5b12473d-3d7d-46fb-ae16-569af651e5da" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487823 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b12473d-3d7d-46fb-ae16-569af651e5da" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487832 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487836 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487842 5778 cpu_manager.go:414] "containerMap: removing container" podUID="5efec5dc-2357-4a61-9fe5-3e73b516be01" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487861 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efec5dc-2357-4a61-9fe5-3e73b516be01" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487868 5778 cpu_manager.go:414] "containerMap: removing container" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487873 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487881 5778 cpu_manager.go:414] "containerMap: removing container" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487886 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487892 5778 cpu_manager.go:414] "containerMap: removing container" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487896 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487902 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487907 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487914 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487918 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487923 5778 cpu_manager.go:414] "containerMap: removing container" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487928 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487933 5778 cpu_manager.go:414] "containerMap: removing container" podUID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487937 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487945 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487949 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487957 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487961 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487970 5778 cpu_manager.go:414] "containerMap: removing container" podUID="9eae91fb-897c-4288-ac57-86173fe4e051" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487975 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eae91fb-897c-4288-ac57-86173fe4e051" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487985 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487989 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.487998 5778 cpu_manager.go:414] "containerMap: removing container" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488003 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488102 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdd665f-021d-4fa4-a753-d0051b31c6d8" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488114 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efec5dc-2357-4a61-9fe5-3e73b516be01" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488125 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488132 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488140 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7ae301-5191-4511-912d-fb6bdb280530" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488149 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c560e2a1-64d9-4270-ba56-bd2217f2a687" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488158 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488165 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="405525e1-ba54-4275-b2f1-7c31e8f88f82" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488172 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b256b1-7cc1-49a1-b66f-27a006b6f5bd" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488181 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488188 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e0b1bb-04d2-4bc0-9f22-97fdfb2b59bd" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488197 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488203 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cce4af7-41f1-46ef-a412-08a0203f4c5f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488212 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eae91fb-897c-4288-ac57-86173fe4e051" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488223 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9d05f7-c83d-4432-a138-5070de7f3424" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488230 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf7feda-0a0f-4cbe-ac5e-cdbf32902285" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488239 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddf4019-13be-4213-937a-78ef85850d51" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488248 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="6688b26c-1109-45d1-b84b-71d69646e29d" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488257 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1759788-2268-4a5d-9c80-ec6a2b39d528" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488265 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5e8124-afe2-4e38-b3e1-4e63cb2a7aba" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488273 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="affc590b-ffd9-44a5-bba1-91c84c67cccb" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488280 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b1296e-ee30-4ccb-8f99-96a3552178c9" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488286 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="db312c02-f1b1-4eb0-809a-3bf6bdef2776" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488292 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc550de-06fb-4a8f-817b-407ee81dcdaf" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488297 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce72afa2-7f2e-43f2-98fa-6d38973ada6b" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488302 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc37c643-dc48-4601-b18a-acd7262df3f3" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488308 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a753a9-2199-49ce-a1e2-3292ec66f1ab" containerName="payload-pre-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488313 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98c93a2-a15f-4e57-bd94-e5b685a879d4" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488319 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="d448a01a-e368-4968-8912-9d8b1cb19f8f" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488324 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c478da01-6384-454f-b3ce-e521652a51ff" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488329 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a429d996-5732-4237-878a-cb5c498a7570" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488337 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe315b0-6760-49d9-b333-5531f0c66cad" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488343 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="313d013a-33f0-41b6-a92c-e8ec8f58b41d" containerName="payload-processing" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488347 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488353 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b50e76-eec9-41fa-8a9d-c708e67ac7a2" containerName="main" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488359 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c149a45-0908-430e-9f23-c3dabb1b9ea7" containerName="maas-api" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488364 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ffb239-7a92-4b4a-bd57-30c90c3d2942" containerName="istio-proxy" Nov 01 23:13:20.488375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.488369 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b12473d-3d7d-46fb-ae16-569af651e5da" containerName="main" Nov 01 23:13:20.492582 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.491511 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.504486 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.504464 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9"] Nov 01 23:13:20.567249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567210 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567252 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567337 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567371 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1080d5b9-3db7-4753-a38c-9293c8c53b63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567425 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567403 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567467 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567575 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567534 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567589 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6mh\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-kube-api-access-hl6mh\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.567682 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.567665 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668476 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668448 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668485 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668503 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1080d5b9-3db7-4753-a38c-9293c8c53b63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668540 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668567 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668596 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668643 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6mh\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-kube-api-access-hl6mh\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668674 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.668814 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668744 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.669058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.668937 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.669058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.669045 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.669168 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.669155 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.669221 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.669196 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.669388 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.669364 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1080d5b9-3db7-4753-a38c-9293c8c53b63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.671285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.671266 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.671556 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.671533 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.676929 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.676899 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6mh\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-kube-api-access-hl6mh\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.677019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.676997 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1080d5b9-3db7-4753-a38c-9293c8c53b63-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qndz9\" (UID: \"1080d5b9-3db7-4753-a38c-9293c8c53b63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:20.808637 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:20.808585 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:21.086768 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.086711 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9"] Nov 01 23:13:21.089201 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:13:21.089171 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1080d5b9_3db7_4753_a38c_9293c8c53b63.slice/crio-9adf66c87dcbb249a881501b329b851241b0b0e8b356658afa9b9433dcfa5f92 WatchSource:0}: Error finding container 9adf66c87dcbb249a881501b329b851241b0b0e8b356658afa9b9433dcfa5f92: Status 404 returned error can't find the container with id 9adf66c87dcbb249a881501b329b851241b0b0e8b356658afa9b9433dcfa5f92 Nov 01 23:13:21.091418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.091381 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:21.091542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.091451 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:21.091542 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.091483 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:21.217563 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.217522 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" event={"ID":"1080d5b9-3db7-4753-a38c-9293c8c53b63","Type":"ContainerStarted","Data":"f7ac09c46f408f3aa673c8b2ebb5ebd4dd72b4c56088d044a4722b0b578cac65"} Nov 01 23:13:21.217744 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.217578 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" event={"ID":"1080d5b9-3db7-4753-a38c-9293c8c53b63","Type":"ContainerStarted","Data":"9adf66c87dcbb249a881501b329b851241b0b0e8b356658afa9b9433dcfa5f92"} Nov 01 23:13:21.238966 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.238913 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podStartSLOduration=1.238845282 podStartE2EDuration="1.238845282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:21.237475605 +0000 UTC m=+2971.013462434" watchObservedRunningTime="2026-11-01 23:13:21.238845282 +0000 UTC m=+2971.014832185" Nov 01 23:13:21.808826 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:21.808792 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:22.811204 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:22.811169 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:22.811629 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:22.811221 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:23.809283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:23.809245 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:23.809454 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:23.809300 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:24.808961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:24.808926 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:24.809373 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:24.808981 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:25.809147 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:25.809115 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:25.809554 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:25.809168 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:26.809652 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:26.809618 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:26.810120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:26.809673 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:27.809215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:27.809174 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 23:13:27.809432 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:27.809236 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 23:13:29.809951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:29.809844 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:29.809951 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:29.809927 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:30.852034 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:30.851986 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:30.852581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:30.852040 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:31.150120 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:31.150038 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:31.150432 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:31.150409 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:31.151239 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:31.151220 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" Nov 01 23:13:35.511371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.511332 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr"] Nov 01 23:13:35.511998 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.511977 5778 cpu_manager.go:414] "containerMap: removing container" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:35.512060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.512001 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:35.512060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.512037 5778 cpu_manager.go:414] "containerMap: removing container" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:35.512060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.512045 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7520856-7bc0-4cce-a430-f537fe2e800d" containerName="istio-proxy" Nov 01 23:13:35.512211 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.512196 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6109756-fe25-4c5e-8ed2-0d42ad4bc9e1" containerName="istio-proxy" Nov 01 23:13:35.512251 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.512217 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d157576-5447-4574-8903-4cc537809016" containerName="istio-proxy" Nov 01 23:13:35.516003 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.515983 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.535220 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.528201 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr"] Nov 01 23:13:35.616441 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616411 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72936583-7f29-479e-a06a-d584522d03e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616453 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616472 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616627 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616570 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616670 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616721 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72936583-7f29-479e-a06a-d584522d03e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616827 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616873 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.616925 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.616896 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpvp\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-kube-api-access-hwpvp\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717824 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717791 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717829 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717875 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpvp\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-kube-api-access-hwpvp\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717904 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72936583-7f29-479e-a06a-d584522d03e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717948 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.717996 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.717973 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718280 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718018 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718280 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718076 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718280 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718214 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72936583-7f29-479e-a06a-d584522d03e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718358 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718446 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718391 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718518 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718481 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718561 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718519 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.718841 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.718821 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72936583-7f29-479e-a06a-d584522d03e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.720375 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.720350 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72936583-7f29-479e-a06a-d584522d03e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.720663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.720643 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72936583-7f29-479e-a06a-d584522d03e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.725645 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.725613 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.725935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.725915 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpvp\" (UniqueName: \"kubernetes.io/projected/72936583-7f29-479e-a06a-d584522d03e0-kube-api-access-hwpvp\") pod \"maas-default-gateway-openshift-default-6886b96f9-hngcr\" (UID: \"72936583-7f29-479e-a06a-d584522d03e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:35.834389 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:35.834332 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:36.111191 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.111105 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr"] Nov 01 23:13:36.114279 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:13:36.114240 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72936583_7f29_479e_a06a_d584522d03e0.slice/crio-ed1a83fe4f4ef0078e875f2a4bffbe1bcc07e055bf931bc16b93f91067c73f05 WatchSource:0}: Error finding container ed1a83fe4f4ef0078e875f2a4bffbe1bcc07e055bf931bc16b93f91067c73f05: Status 404 returned error can't find the container with id ed1a83fe4f4ef0078e875f2a4bffbe1bcc07e055bf931bc16b93f91067c73f05 Nov 01 23:13:36.116507 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.116475 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:36.116572 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.116544 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:36.116607 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.116570 5778 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 23:13:36.323371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.323330 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" event={"ID":"72936583-7f29-479e-a06a-d584522d03e0","Type":"ContainerStarted","Data":"da2a097bf38e8396a3e1c5c686dcfed58309db403e7bf79a7d44e1dd333a3fcd"} Nov 01 23:13:36.323546 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.323400 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" event={"ID":"72936583-7f29-479e-a06a-d584522d03e0","Type":"ContainerStarted","Data":"ed1a83fe4f4ef0078e875f2a4bffbe1bcc07e055bf931bc16b93f91067c73f05"} Nov 01 23:13:36.345164 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.345118 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podStartSLOduration=1.34510427 podStartE2EDuration="1.34510427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:36.344408654 +0000 UTC m=+2986.120395495" watchObservedRunningTime="2026-11-01 23:13:36.34510427 +0000 UTC m=+2986.121091096" Nov 01 23:13:36.839905 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:36.835948 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:37.840027 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:37.839984 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:37.840577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:37.840048 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:38.835320 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:38.835277 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:38.835625 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:38.835594 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:39.835257 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:39.835217 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:39.836126 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:39.836092 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:40.837628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:40.837180 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:40.837628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:40.837241 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:41.838877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:41.837945 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:41.838877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:41.838006 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:42.835370 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:42.835331 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 23:13:42.835551 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:42.835389 5778 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 23:13:44.245492 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:44.245445 5778 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:44.245989 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:44.245884 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:44.246766 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:44.246740 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" Nov 01 23:13:48.772702 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:13:48.772660 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:13:50.937163 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:50.937134 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:13:51.000185 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:13:51.000157 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:14:18.772689 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:14:18.772654 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:14:30.292321 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:14:30.292282 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:30.294966 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:14:30.292833 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:48.890911 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:14:48.890816 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:14:57.141466 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:14:57.141429 5778 scope.go:122] "RemoveContainer" containerID="9781e802180009aa35d5e94fea685c1e4d5c0b35a6b01e8987842a10f7365f5d" Nov 01 23:15:18.772139 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:15:18.772100 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:15:48.772546 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:15:48.772509 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:16:15.298008 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:16:15.297968 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:16:15.298707 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:16:15.298671 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:16:18.773557 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:16:18.773519 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:16:48.774904 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:16:48.774844 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:17:02.210004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:02.209966 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:17:02.210543 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:02.210035 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:17:15.051796 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.051744 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 23:17:15.052370 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.051998 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" podUID="8a148739-f6ef-4959-b862-beb7b3ea7aea" containerName="discovery" containerID="cri-o://4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8" gracePeriod=30 Nov 01 23:17:15.329631 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.329606 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 23:17:15.427216 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427180 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427229 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427257 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427299 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427395 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427441 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427663 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427509 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\") pod \"8a148739-f6ef-4959-b862-beb7b3ea7aea\" (UID: \"8a148739-f6ef-4959-b862-beb7b3ea7aea\") " Nov 01 23:17:15.427778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427752 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:17:15.428039 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.427994 5778 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-ca-configmap\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.430641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.430584 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:15.430641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.430597 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:15.430641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.430606 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:17:15.431071 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.431049 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "kube-api-access-t92dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:17:15.431313 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.431289 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:17:15.431429 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.431390 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert" pod "8a148739-f6ef-4959-b862-beb7b3ea7aea" (UID: "8a148739-f6ef-4959-b862-beb7b3ea7aea"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:15.529101 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529070 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92dc\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-kube-api-access-t92dc\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.529101 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529096 5778 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-csr-dns-cert\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.529101 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529107 5778 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-kubeconfig\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.529326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529116 5778 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a148739-f6ef-4959-b862-beb7b3ea7aea-istio-token\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.529326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529124 5778 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a148739-f6ef-4959-b862-beb7b3ea7aea-cacerts\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.529326 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.529134 5778 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a148739-f6ef-4959-b862-beb7b3ea7aea-local-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:15.792025 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.791980 5778 generic.go:356] "Generic (PLEG): container finished" podID="8a148739-f6ef-4959-b862-beb7b3ea7aea" containerID="4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8" exitCode=0 Nov 01 23:17:15.792248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.792067 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" Nov 01 23:17:15.792248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.792068 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" event={"ID":"8a148739-f6ef-4959-b862-beb7b3ea7aea","Type":"ContainerDied","Data":"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8"} Nov 01 23:17:15.792248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.792158 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4" event={"ID":"8a148739-f6ef-4959-b862-beb7b3ea7aea","Type":"ContainerDied","Data":"677dd39f01a093b8bdc7b838d4b7b3d24ea868fd049bb453ab7fe0cc875315f5"} Nov 01 23:17:15.792248 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.792186 5778 scope.go:122] "RemoveContainer" containerID="4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8" Nov 01 23:17:15.806282 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.806264 5778 scope.go:122] "RemoveContainer" containerID="4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8" Nov 01 23:17:15.806521 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:17:15.806503 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8\": container with ID starting with 4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8 not found: ID does not exist" containerID="4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8" Nov 01 23:17:15.806603 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.806529 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8"} err="failed to get container status \"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8\": rpc error: code = NotFound desc = could not find container \"4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8\": container with ID starting with 4b91fcbbc79e473313ac36dff59f0ec961782323a005b01cb029f1c181e831f8 not found: ID does not exist" Nov 01 23:17:15.819051 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.819029 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 23:17:15.822827 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:15.822803 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8lx4"] Nov 01 23:17:16.372255 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:17:16.372224 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a148739-f6ef-4959-b862-beb7b3ea7aea" path="/var/lib/kubelet/pods/8a148739-f6ef-4959-b862-beb7b3ea7aea/volumes" Nov 01 23:17:18.773675 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:17:18.773639 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:17:48.772048 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:17:48.772004 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:18:10.841580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:18:10.841042 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bftkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:18:10.841580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:18:10.841110 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bftkj" podUID="e6e8ec16-0274-4d53-8fcf-bc8df04dbed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:18:18.773358 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:18:18.773324 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:18:48.773141 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:18:48.773103 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:18:50.972325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:18:50.972299 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:18:51.041924 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:18:51.041897 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:19:18.772598 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:19:18.772556 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:19:48.869878 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:19:48.867092 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:19:54.435761 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.435725 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 23:19:54.436236 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.435953 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" containerID="cri-o://3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66" gracePeriod=10 Nov 01 23:19:54.700339 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.700283 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 23:19:54.735888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.735286 5778 generic.go:356] "Generic (PLEG): container finished" podID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerID="3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66" exitCode=0 Nov 01 23:19:54.735888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.735334 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerDied","Data":"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66"} Nov 01 23:19:54.735888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.735350 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" Nov 01 23:19:54.735888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.735386 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p" event={"ID":"c72b62f8-fa79-418a-a5fe-e92fa199498a","Type":"ContainerDied","Data":"69fc897cd4697bb823bcdf790e227c59211c34a32eeaba5381ecd91a0d26993d"} Nov 01 23:19:54.735888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.735418 5778 scope.go:122] "RemoveContainer" containerID="3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66" Nov 01 23:19:54.747157 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.747124 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\") pod \"c72b62f8-fa79-418a-a5fe-e92fa199498a\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " Nov 01 23:19:54.747338 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.747318 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\" (UniqueName: \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\") pod \"c72b62f8-fa79-418a-a5fe-e92fa199498a\" (UID: \"c72b62f8-fa79-418a-a5fe-e92fa199498a\") " Nov 01 23:19:54.747485 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.747462 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume" pod "c72b62f8-fa79-418a-a5fe-e92fa199498a" (UID: "c72b62f8-fa79-418a-a5fe-e92fa199498a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:54.747641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.747618 5778 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c72b62f8-fa79-418a-a5fe-e92fa199498a-extensions-socket-volume\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:54.749601 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.749579 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2" pod "c72b62f8-fa79-418a-a5fe-e92fa199498a" (UID: "c72b62f8-fa79-418a-a5fe-e92fa199498a"). InnerVolumeSpecName "kube-api-access-tbmx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:54.751944 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.751917 5778 scope.go:122] "RemoveContainer" containerID="23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377" Nov 01 23:19:54.762325 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.762298 5778 scope.go:122] "RemoveContainer" containerID="3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66" Nov 01 23:19:54.762702 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:19:54.762680 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66\": container with ID starting with 3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66 not found: ID does not exist" containerID="3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66" Nov 01 23:19:54.762811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.762712 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66"} err="failed to get container status \"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66\": rpc error: code = NotFound desc = could not find container \"3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66\": container with ID starting with 3db74e301564e34a1760613c434ddd81cd6ad721ccaabfa35778efae61e2bc66 not found: ID does not exist" Nov 01 23:19:54.762811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.762731 5778 scope.go:122] "RemoveContainer" containerID="23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377" Nov 01 23:19:54.763051 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:19:54.763024 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377\": container with ID starting with 23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377 not found: ID does not exist" containerID="23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377" Nov 01 23:19:54.763131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.763059 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377"} err="failed to get container status \"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377\": rpc error: code = NotFound desc = could not find container \"23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377\": container with ID starting with 23b96fbdc0bd73c268f93f62dc51525d0b0f068900163f3d6906c6e570788377 not found: ID does not exist" Nov 01 23:19:54.848482 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:54.848448 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmx2\" (UniqueName: \"kubernetes.io/projected/c72b62f8-fa79-418a-a5fe-e92fa199498a-kube-api-access-tbmx2\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:55.061559 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:55.061529 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 23:19:55.065942 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:55.065916 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c2f4p"] Nov 01 23:19:56.373528 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:19:56.373494 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" path="/var/lib/kubelet/pods/c72b62f8-fa79-418a-a5fe-e92fa199498a/volumes" Nov 01 23:20:18.771582 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:20:18.771546 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:20:48.772291 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:20:48.772247 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:20:55.442182 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442146 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99"] Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442536 5778 cpu_manager.go:414] "containerMap: removing container" podUID="8a148739-f6ef-4959-b862-beb7b3ea7aea" containerName="discovery" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442547 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a148739-f6ef-4959-b862-beb7b3ea7aea" containerName="discovery" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442563 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442568 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442577 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442582 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442640 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442649 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72b62f8-fa79-418a-a5fe-e92fa199498a" containerName="manager" Nov 01 23:20:55.442900 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.442660 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a148739-f6ef-4959-b862-beb7b3ea7aea" containerName="discovery" Nov 01 23:20:55.445777 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.445758 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.452783 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.452402 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lvjbk\"" Nov 01 23:20:55.457782 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.457761 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsw6f\" (UniqueName: \"kubernetes.io/projected/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-kube-api-access-nsw6f\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.457939 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.457814 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.458272 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.458251 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99"] Nov 01 23:20:55.558937 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.558913 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsw6f\" (UniqueName: \"kubernetes.io/projected/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-kube-api-access-nsw6f\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.559060 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.558957 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.559319 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.559302 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.567826 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.567800 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsw6f\" (UniqueName: \"kubernetes.io/projected/2681e5ba-efcc-4beb-bdfd-3cbfe1d30983-kube-api-access-nsw6f\") pod \"kuadrant-operator-controller-manager-678d87766f-85s99\" (UID: \"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:55.759924 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:55.759846 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:56.040723 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:56.040693 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99"] Nov 01 23:20:56.042438 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:20:56.042410 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2681e5ba_efcc_4beb_bdfd_3cbfe1d30983.slice/crio-095a36a6f01a1e2690627b653942585168f69487d9df0484b84a6c62de96c986 WatchSource:0}: Error finding container 095a36a6f01a1e2690627b653942585168f69487d9df0484b84a6c62de96c986: Status 404 returned error can't find the container with id 095a36a6f01a1e2690627b653942585168f69487d9df0484b84a6c62de96c986 Nov 01 23:20:56.052537 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:56.052511 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" event={"ID":"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983","Type":"ContainerStarted","Data":"095a36a6f01a1e2690627b653942585168f69487d9df0484b84a6c62de96c986"} Nov 01 23:20:57.058862 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:57.058826 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" event={"ID":"2681e5ba-efcc-4beb-bdfd-3cbfe1d30983","Type":"ContainerStarted","Data":"f17b6a1e03c1e3bae8a2552d015441e6295a8acdd16e7f4adeb8223687022c99"} Nov 01 23:20:57.059283 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:57.058905 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:20:57.085472 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:20:57.085426 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" podStartSLOduration=2.08541307 podStartE2EDuration="2.08541307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:20:57.082784356 +0000 UTC m=+3426.858771185" watchObservedRunningTime="2026-11-01 23:20:57.08541307 +0000 UTC m=+3426.861399900" Nov 01 23:21:08.066293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:21:08.066218 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85s99" Nov 01 23:21:18.773877 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:21:18.773825 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:21:48.774060 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:21:48.774001 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:22:18.772547 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:22:18.772508 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:22:48.794303 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:22:48.794260 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:23:15.297906 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:15.294961 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hngcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:15.297906 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:15.295040 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hngcr" podUID="72936583-7f29-479e-a06a-d584522d03e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:18.772376 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:23:18.772337 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:23:32.204056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:32.204004 5778 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qndz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:32.204675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:32.204643 5778 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qndz9" podUID="1080d5b9-3db7-4753-a38c-9293c8c53b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:48.772516 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:23:48.772479 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:23:51.010094 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:51.010067 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:23:51.080170 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:23:51.080143 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:24:18.772552 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:24:18.772511 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:24:48.423285 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:48.423248 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 23:24:48.423935 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:48.423508 5778 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" podUID="b1939254-2eef-4572-a8e1-6d075ab39eb9" containerName="maas-api" containerID="cri-o://3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762" gracePeriod=30 Nov 01 23:24:48.773258 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:24:48.773173 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:24:49.691477 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.691453 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 23:24:49.697023 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.696996 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\") pod \"b1939254-2eef-4572-a8e1-6d075ab39eb9\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " Nov 01 23:24:49.697129 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.697107 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\") pod \"b1939254-2eef-4572-a8e1-6d075ab39eb9\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " Nov 01 23:24:49.697176 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.697138 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\" (UniqueName: \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\") pod \"b1939254-2eef-4572-a8e1-6d075ab39eb9\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " Nov 01 23:24:49.697231 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.697179 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\") pod \"b1939254-2eef-4572-a8e1-6d075ab39eb9\" (UID: \"b1939254-2eef-4572-a8e1-6d075ab39eb9\") " Nov 01 23:24:49.697473 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.697445 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca" pod "b1939254-2eef-4572-a8e1-6d075ab39eb9" (UID: "b1939254-2eef-4572-a8e1-6d075ab39eb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:24:49.697577 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.697559 5778 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1939254-2eef-4572-a8e1-6d075ab39eb9-openshift-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:49.699348 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.699321 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls" pod "b1939254-2eef-4572-a8e1-6d075ab39eb9" (UID: "b1939254-2eef-4572-a8e1-6d075ab39eb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:49.699438 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.699375 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs" pod "b1939254-2eef-4572-a8e1-6d075ab39eb9" (UID: "b1939254-2eef-4572-a8e1-6d075ab39eb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:49.699438 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.699381 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf" pod "b1939254-2eef-4572-a8e1-6d075ab39eb9" (UID: "b1939254-2eef-4572-a8e1-6d075ab39eb9"). InnerVolumeSpecName "kube-api-access-7kxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:49.798317 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.798226 5778 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-metrics-certs\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:49.798317 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.798262 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxqf\" (UniqueName: \"kubernetes.io/projected/b1939254-2eef-4572-a8e1-6d075ab39eb9-kube-api-access-7kxqf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:49.798317 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:49.798276 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1939254-2eef-4572-a8e1-6d075ab39eb9-maas-api-tls\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:50.490284 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.490252 5778 generic.go:356] "Generic (PLEG): container finished" podID="b1939254-2eef-4572-a8e1-6d075ab39eb9" containerID="3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762" exitCode=0 Nov 01 23:24:50.490470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.490306 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" Nov 01 23:24:50.490470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.490325 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" event={"ID":"b1939254-2eef-4572-a8e1-6d075ab39eb9","Type":"ContainerDied","Data":"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762"} Nov 01 23:24:50.490470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.490367 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-lbw44" event={"ID":"b1939254-2eef-4572-a8e1-6d075ab39eb9","Type":"ContainerDied","Data":"978fa81dfd8f4de7cc36dc9b32de5da7f246c962d952d3497a77bc69991a9613"} Nov 01 23:24:50.490470 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.490393 5778 scope.go:122] "RemoveContainer" containerID="3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762" Nov 01 23:24:50.501888 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.501836 5778 scope.go:122] "RemoveContainer" containerID="3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762" Nov 01 23:24:50.502172 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:24:50.502143 5778 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762\": container with ID starting with 3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762 not found: ID does not exist" containerID="3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762" Nov 01 23:24:50.502270 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.502181 5778 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762"} err="failed to get container status \"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762\": rpc error: code = NotFound desc = could not find container \"3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762\": container with ID starting with 3083f9be0abf5ecb3f13bef0989a69c3e35a18dc8d9e089bfd9cfc8b7a685762 not found: ID does not exist" Nov 01 23:24:50.508042 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.508015 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 23:24:50.510269 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:50.510249 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-lbw44"] Nov 01 23:24:52.372549 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:52.372512 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1939254-2eef-4572-a8e1-6d075ab39eb9" path="/var/lib/kubelet/pods/b1939254-2eef-4572-a8e1-6d075ab39eb9/volumes" Nov 01 23:24:54.586376 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.586341 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm"] Nov 01 23:24:54.586860 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.586789 5778 cpu_manager.go:414] "containerMap: removing container" podUID="b1939254-2eef-4572-a8e1-6d075ab39eb9" containerName="maas-api" Nov 01 23:24:54.586860 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.586800 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1939254-2eef-4572-a8e1-6d075ab39eb9" containerName="maas-api" Nov 01 23:24:54.586964 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.586903 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1939254-2eef-4572-a8e1-6d075ab39eb9" containerName="maas-api" Nov 01 23:24:54.590200 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.590183 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.594939 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.594914 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:24:54.596080 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.596048 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:24:54.598606 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.598582 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm"] Nov 01 23:24:54.642700 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.642662 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.642911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.642720 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zd5\" (UniqueName: \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.642911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.642841 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.744023 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.743976 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.744318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.744078 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.744318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.744108 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zd5\" (UniqueName: \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.744694 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.744670 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.746831 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.746797 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.755344 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.755324 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zd5\" (UniqueName: \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\") pod \"maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:54.905930 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:54.905907 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:55.183506 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:55.183477 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm"] Nov 01 23:24:55.185236 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:24:55.185197 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b56bd0_8d41_4073_9814_d7c6a172baac.slice/crio-97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb WatchSource:0}: Error finding container 97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb: Status 404 returned error can't find the container with id 97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb Nov 01 23:24:55.529433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:55.529357 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" event={"ID":"72b56bd0-8d41-4073-9814-d7c6a172baac","Type":"ContainerStarted","Data":"9fef477a0511fbfb39089a601757d8186a27544e934e7818c5503859acad0f0c"} Nov 01 23:24:55.529433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:55.529409 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" event={"ID":"72b56bd0-8d41-4073-9814-d7c6a172baac","Type":"ContainerStarted","Data":"97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb"} Nov 01 23:24:55.544091 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:55.544040 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" podStartSLOduration=1.544024425 podStartE2EDuration="1.544024425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:55.543689192 +0000 UTC m=+3665.319676022" watchObservedRunningTime="2026-11-01 23:24:55.544024425 +0000 UTC m=+3665.320011245" Nov 01 23:24:56.537318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:56.537286 5778 generic.go:356] "Generic (PLEG): container finished" podID="72b56bd0-8d41-4073-9814-d7c6a172baac" containerID="9fef477a0511fbfb39089a601757d8186a27544e934e7818c5503859acad0f0c" exitCode=0 Nov 01 23:24:56.537740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:56.537338 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" event={"ID":"72b56bd0-8d41-4073-9814-d7c6a172baac","Type":"ContainerDied","Data":"9fef477a0511fbfb39089a601757d8186a27544e934e7818c5503859acad0f0c"} Nov 01 23:24:57.689579 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.689550 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:57.775048 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.775013 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\") pod \"72b56bd0-8d41-4073-9814-d7c6a172baac\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " Nov 01 23:24:57.775048 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.775052 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\") pod \"72b56bd0-8d41-4073-9814-d7c6a172baac\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " Nov 01 23:24:57.775278 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.775170 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\" (UniqueName: \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\") pod \"72b56bd0-8d41-4073-9814-d7c6a172baac\" (UID: \"72b56bd0-8d41-4073-9814-d7c6a172baac\") " Nov 01 23:24:57.775413 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.775391 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca" pod "72b56bd0-8d41-4073-9814-d7c6a172baac" (UID: "72b56bd0-8d41-4073-9814-d7c6a172baac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:24:57.775538 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.775512 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:57.777459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.777432 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5" pod "72b56bd0-8d41-4073-9814-d7c6a172baac" (UID: "72b56bd0-8d41-4073-9814-d7c6a172baac"). InnerVolumeSpecName "kube-api-access-z4zd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:57.777459 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.777440 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca" pod "72b56bd0-8d41-4073-9814-d7c6a172baac" (UID: "72b56bd0-8d41-4073-9814-d7c6a172baac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:57.876360 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.876266 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zd5\" (UniqueName: \"kubernetes.io/projected/72b56bd0-8d41-4073-9814-d7c6a172baac-kube-api-access-z4zd5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:57.876360 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:57.876300 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b56bd0-8d41-4073-9814-d7c6a172baac-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:58.548669 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:58.548638 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" Nov 01 23:24:58.548887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:58.548669 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm" event={"ID":"72b56bd0-8d41-4073-9814-d7c6a172baac","Type":"ContainerDied","Data":"97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb"} Nov 01 23:24:58.548887 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:24:58.548724 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97149b0d799e257788668975be5d66553718fae45cc7cb3c0d50b565ee1807fb" Nov 01 23:25:04.766324 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:25:04.766278 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm"] Nov 01 23:25:04.768727 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:25:04.768701 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-417a41eb-8q5vm"] Nov 01 23:25:06.374503 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:25:06.374465 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b56bd0-8d41-4073-9814-d7c6a172baac" path="/var/lib/kubelet/pods/72b56bd0-8d41-4073-9814-d7c6a172baac/volumes" Nov 01 23:25:18.775305 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:25:18.775253 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:25:48.774698 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:25:48.774656 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:26:18.772577 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:26:18.772537 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:26:48.772774 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:26:48.772732 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:26:52.595906 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.595868 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc"] Nov 01 23:26:52.596336 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.596265 5778 cpu_manager.go:414] "containerMap: removing container" podUID="72b56bd0-8d41-4073-9814-d7c6a172baac" containerName="revoke-keys" Nov 01 23:26:52.596336 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.596277 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b56bd0-8d41-4073-9814-d7c6a172baac" containerName="revoke-keys" Nov 01 23:26:52.596421 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.596361 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b56bd0-8d41-4073-9814-d7c6a172baac" containerName="revoke-keys" Nov 01 23:26:52.599511 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.599486 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.604206 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.604186 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:26:52.604302 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.604231 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:26:52.610682 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.610656 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc"] Nov 01 23:26:52.717012 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.716975 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72bf\" (UniqueName: \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.717200 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.717027 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.717200 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.717156 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.818505 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.818464 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72bf\" (UniqueName: \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.818709 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.818514 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.818709 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.818572 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.819276 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.819251 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.821255 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.821236 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.827059 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.827036 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72bf\" (UniqueName: \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:52.914834 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:52.914796 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:53.192193 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:53.192165 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc"] Nov 01 23:26:53.194585 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:26:53.194556 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c1f6fa_2f40_417e_83e1_9d60c3d4c9d3.slice/crio-fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990 WatchSource:0}: Error finding container fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990: Status 404 returned error can't find the container with id fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990 Nov 01 23:26:54.167683 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:54.167653 5778 generic.go:356] "Generic (PLEG): container finished" podID="c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" containerID="4ea1658c15b6fbcf0a51ddd465461152778f084f16e249300b4bc73aabb20e8e" exitCode=0 Nov 01 23:26:54.168112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:54.167726 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" event={"ID":"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3","Type":"ContainerDied","Data":"4ea1658c15b6fbcf0a51ddd465461152778f084f16e249300b4bc73aabb20e8e"} Nov 01 23:26:54.168112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:54.167799 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" event={"ID":"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3","Type":"ContainerStarted","Data":"fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990"} Nov 01 23:26:55.309062 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.309038 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:55.445678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.445585 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\") pod \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " Nov 01 23:26:55.445678 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.445638 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\") pod \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " Nov 01 23:26:55.445983 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.445733 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\" (UniqueName: \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\") pod \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\" (UID: \"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3\") " Nov 01 23:26:55.446099 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.446074 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca" pod "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" (UID: "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:26:55.448006 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.447981 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca" pod "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" (UID: "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:26:55.448111 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.448095 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf" pod "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" (UID: "c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3"). InnerVolumeSpecName "kube-api-access-r72bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:26:55.547351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.547318 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:26:55.547351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.547346 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:26:55.547351 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:55.547356 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72bf\" (UniqueName: \"kubernetes.io/projected/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3-kube-api-access-r72bf\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:26:56.180237 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:56.180210 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" Nov 01 23:26:56.180426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:56.180238 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc" event={"ID":"c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3","Type":"ContainerDied","Data":"fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990"} Nov 01 23:26:56.180426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:26:56.180286 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed220faa266cc188c184467f3232c5c22a5f95d6542ed91bfaf40bc55f25990" Nov 01 23:27:08.365702 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:08.365598 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc"] Nov 01 23:27:08.367738 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:08.367717 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ede84d-gwmdc"] Nov 01 23:27:08.374047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:08.374018 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" path="/var/lib/kubelet/pods/c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3/volumes" Nov 01 23:27:18.881120 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:27:18.881076 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:27:26.221660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.221622 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6"] Nov 01 23:27:26.222272 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.222253 5778 cpu_manager.go:414] "containerMap: removing container" podUID="c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" containerName="revoke-keys" Nov 01 23:27:26.222341 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.222274 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" containerName="revoke-keys" Nov 01 23:27:26.222383 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.222342 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c1f6fa-2f40-417e-83e1-9d60c3d4c9d3" containerName="revoke-keys" Nov 01 23:27:26.225407 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.225387 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.231196 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.231171 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:27:26.231333 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.231270 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:27:26.234296 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.234271 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6"] Nov 01 23:27:26.309191 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.309159 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd4l9\" (UniqueName: \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.309347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.309224 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.309347 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.309306 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.409961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.409926 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.409961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.409963 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.410199 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.410065 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd4l9\" (UniqueName: \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.410596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.410573 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.412493 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.412471 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.418250 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.418231 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd4l9\" (UniqueName: \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\") pod \"maas-api-revoke-sub-f03629db-lmbk6\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.540534 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.540466 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:26.812967 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:26.812938 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6"] Nov 01 23:27:26.815204 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:27:26.815169 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08277ab_2d20_4dc3_b737_5420b2e0425b.slice/crio-1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0 WatchSource:0}: Error finding container 1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0: Status 404 returned error can't find the container with id 1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0 Nov 01 23:27:27.402641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:27.402562 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" event={"ID":"a08277ab-2d20-4dc3-b737-5420b2e0425b","Type":"ContainerStarted","Data":"ab2e7a0f17a7437bb8bc19ba0e844e819cfde1fcb126a6b2ca0789dbde518327"} Nov 01 23:27:27.402641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:27.402624 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" event={"ID":"a08277ab-2d20-4dc3-b737-5420b2e0425b","Type":"ContainerStarted","Data":"1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0"} Nov 01 23:27:27.419222 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:27.419176 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" podStartSLOduration=1.419161482 podStartE2EDuration="1.419161482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:27:27.418007545 +0000 UTC m=+3817.193994375" watchObservedRunningTime="2026-11-01 23:27:27.419161482 +0000 UTC m=+3817.195148307" Nov 01 23:27:28.409165 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:28.409136 5778 generic.go:356] "Generic (PLEG): container finished" podID="a08277ab-2d20-4dc3-b737-5420b2e0425b" containerID="ab2e7a0f17a7437bb8bc19ba0e844e819cfde1fcb126a6b2ca0789dbde518327" exitCode=0 Nov 01 23:27:28.409581 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:28.409229 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" event={"ID":"a08277ab-2d20-4dc3-b737-5420b2e0425b","Type":"ContainerDied","Data":"ab2e7a0f17a7437bb8bc19ba0e844e819cfde1fcb126a6b2ca0789dbde518327"} Nov 01 23:27:29.551275 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.551249 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:29.639293 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.639257 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\") pod \"a08277ab-2d20-4dc3-b737-5420b2e0425b\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " Nov 01 23:27:29.639433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.639351 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\") pod \"a08277ab-2d20-4dc3-b737-5420b2e0425b\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " Nov 01 23:27:29.639433 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.639383 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\" (UniqueName: \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\") pod \"a08277ab-2d20-4dc3-b737-5420b2e0425b\" (UID: \"a08277ab-2d20-4dc3-b737-5420b2e0425b\") " Nov 01 23:27:29.639653 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.639629 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca" pod "a08277ab-2d20-4dc3-b737-5420b2e0425b" (UID: "a08277ab-2d20-4dc3-b737-5420b2e0425b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:29.641665 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.641638 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca" pod "a08277ab-2d20-4dc3-b737-5420b2e0425b" (UID: "a08277ab-2d20-4dc3-b737-5420b2e0425b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:27:29.641781 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.641669 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9" pod "a08277ab-2d20-4dc3-b737-5420b2e0425b" (UID: "a08277ab-2d20-4dc3-b737-5420b2e0425b"). InnerVolumeSpecName "kube-api-access-cd4l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:29.740957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.740899 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:29.740957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.740922 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd4l9\" (UniqueName: \"kubernetes.io/projected/a08277ab-2d20-4dc3-b737-5420b2e0425b-kube-api-access-cd4l9\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:29.740957 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:29.740934 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08277ab-2d20-4dc3-b737-5420b2e0425b-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:30.420691 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:30.420661 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" Nov 01 23:27:30.420844 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:30.420675 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmbk6" event={"ID":"a08277ab-2d20-4dc3-b737-5420b2e0425b","Type":"ContainerDied","Data":"1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0"} Nov 01 23:27:30.420844 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:30.420740 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f546100d27beb44cd9319f168273546b2f0562bae21b7f42d5033dd852af4d0" Nov 01 23:27:42.265032 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.264992 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk"] Nov 01 23:27:42.265531 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.265418 5778 cpu_manager.go:414] "containerMap: removing container" podUID="a08277ab-2d20-4dc3-b737-5420b2e0425b" containerName="revoke-keys" Nov 01 23:27:42.265531 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.265432 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08277ab-2d20-4dc3-b737-5420b2e0425b" containerName="revoke-keys" Nov 01 23:27:42.265531 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.265511 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08277ab-2d20-4dc3-b737-5420b2e0425b" containerName="revoke-keys" Nov 01 23:27:42.268774 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.268757 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.275074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.275044 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:27:42.275074 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.275046 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:27:42.277924 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.277899 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk"] Nov 01 23:27:42.378010 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.377978 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.378173 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.378030 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.378173 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.378124 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcb6c\" (UniqueName: \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.479329 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.479295 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.479481 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.479353 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.479481 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.479385 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcb6c\" (UniqueName: \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.480151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.480129 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.481927 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.481909 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.487378 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.487356 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcb6c\" (UniqueName: \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\") pod \"maas-api-revoke-sub-0165d7d2-f4tjk\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.584201 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.584135 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:42.862620 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:42.862591 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk"] Nov 01 23:27:42.864729 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:27:42.864697 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4993adc2_d160_4b47_b0d3_1d19db148aba.slice/crio-e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa WatchSource:0}: Error finding container e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa: Status 404 returned error can't find the container with id e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa Nov 01 23:27:43.512525 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:43.512474 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" event={"ID":"4993adc2-d160-4b47-b0d3-1d19db148aba","Type":"ContainerStarted","Data":"287227bd6620ab943c21c2bb5ea5f1b70ba6cf8a266cd106aa1e4735e1b71e00"} Nov 01 23:27:43.513023 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:43.512538 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" event={"ID":"4993adc2-d160-4b47-b0d3-1d19db148aba","Type":"ContainerStarted","Data":"e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa"} Nov 01 23:27:43.530580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:43.530522 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" podStartSLOduration=1.530505563 podStartE2EDuration="1.530505563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:27:43.529951681 +0000 UTC m=+3833.305938513" watchObservedRunningTime="2026-11-01 23:27:43.530505563 +0000 UTC m=+3833.306492392" Nov 01 23:27:44.518087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:44.518053 5778 generic.go:356] "Generic (PLEG): container finished" podID="4993adc2-d160-4b47-b0d3-1d19db148aba" containerID="287227bd6620ab943c21c2bb5ea5f1b70ba6cf8a266cd106aa1e4735e1b71e00" exitCode=0 Nov 01 23:27:44.518513 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:44.518141 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" event={"ID":"4993adc2-d160-4b47-b0d3-1d19db148aba","Type":"ContainerDied","Data":"287227bd6620ab943c21c2bb5ea5f1b70ba6cf8a266cd106aa1e4735e1b71e00"} Nov 01 23:27:45.661499 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.661471 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:45.808836 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.808761 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\" (UniqueName: \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\") pod \"4993adc2-d160-4b47-b0d3-1d19db148aba\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " Nov 01 23:27:45.809056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.808839 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\") pod \"4993adc2-d160-4b47-b0d3-1d19db148aba\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " Nov 01 23:27:45.809056 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.808904 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\") pod \"4993adc2-d160-4b47-b0d3-1d19db148aba\" (UID: \"4993adc2-d160-4b47-b0d3-1d19db148aba\") " Nov 01 23:27:45.809288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.809262 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca" pod "4993adc2-d160-4b47-b0d3-1d19db148aba" (UID: "4993adc2-d160-4b47-b0d3-1d19db148aba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:45.811163 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.811139 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca" pod "4993adc2-d160-4b47-b0d3-1d19db148aba" (UID: "4993adc2-d160-4b47-b0d3-1d19db148aba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:27:45.811284 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.811215 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c" pod "4993adc2-d160-4b47-b0d3-1d19db148aba" (UID: "4993adc2-d160-4b47-b0d3-1d19db148aba"). InnerVolumeSpecName "kube-api-access-qcb6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:45.909778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.909751 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcb6c\" (UniqueName: \"kubernetes.io/projected/4993adc2-d160-4b47-b0d3-1d19db148aba-kube-api-access-qcb6c\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:45.909778 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.909775 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:45.909956 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:45.909786 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4993adc2-d160-4b47-b0d3-1d19db148aba-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:46.530628 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:46.530596 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" Nov 01 23:27:46.530807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:46.530587 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f4tjk" event={"ID":"4993adc2-d160-4b47-b0d3-1d19db148aba","Type":"ContainerDied","Data":"e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa"} Nov 01 23:27:46.530807 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:46.530735 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01badd6dbcb7544d9eee7ec5e2264f22dbd624ebf932f6c9b4936e626f9abaa" Nov 01 23:27:48.773042 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:27:48.773008 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:27:52.959484 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.959448 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m"] Nov 01 23:27:52.959959 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.959873 5778 cpu_manager.go:414] "containerMap: removing container" podUID="4993adc2-d160-4b47-b0d3-1d19db148aba" containerName="revoke-keys" Nov 01 23:27:52.959959 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.959890 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4993adc2-d160-4b47-b0d3-1d19db148aba" containerName="revoke-keys" Nov 01 23:27:52.960059 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.960009 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="4993adc2-d160-4b47-b0d3-1d19db148aba" containerName="revoke-keys" Nov 01 23:27:52.963249 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.963223 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:52.968151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.968123 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:27:52.969448 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.969424 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:27:52.972706 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:52.972679 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m"] Nov 01 23:27:53.076395 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.076355 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.076571 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.076406 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8m5\" (UniqueName: \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.076571 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.076511 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.177793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.177753 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.177793 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.177797 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8m5\" (UniqueName: \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.178048 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.177834 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.178554 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.178524 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.180595 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.180566 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.186096 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.186072 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8m5\" (UniqueName: \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\") pod \"maas-api-revoke-sub-6b19338f-z9k7m\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.278536 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.278454 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:53.566171 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:53.565904 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m"] Nov 01 23:27:53.569899 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:27:53.568804 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2418bbb7_dec4_4aca_907d_42279a6945c6.slice/crio-30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0 WatchSource:0}: Error finding container 30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0: Status 404 returned error can't find the container with id 30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0 Nov 01 23:27:54.577754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:54.577720 5778 generic.go:356] "Generic (PLEG): container finished" podID="2418bbb7-dec4-4aca-907d-42279a6945c6" containerID="4d8bb619b142c63bf5d38676c0112fb6c87c2ee8ff0fe24abeee1e061db3cfb8" exitCode=0 Nov 01 23:27:54.578151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:54.577791 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" event={"ID":"2418bbb7-dec4-4aca-907d-42279a6945c6","Type":"ContainerDied","Data":"4d8bb619b142c63bf5d38676c0112fb6c87c2ee8ff0fe24abeee1e061db3cfb8"} Nov 01 23:27:54.578151 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:54.577844 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" event={"ID":"2418bbb7-dec4-4aca-907d-42279a6945c6","Type":"ContainerStarted","Data":"30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0"} Nov 01 23:27:55.719907 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.719839 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:55.902882 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.902798 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\") pod \"2418bbb7-dec4-4aca-907d-42279a6945c6\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " Nov 01 23:27:55.902882 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.902841 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\" (UniqueName: \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\") pod \"2418bbb7-dec4-4aca-907d-42279a6945c6\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " Nov 01 23:27:55.903087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.902894 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\") pod \"2418bbb7-dec4-4aca-907d-42279a6945c6\" (UID: \"2418bbb7-dec4-4aca-907d-42279a6945c6\") " Nov 01 23:27:55.903235 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.903208 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca" pod "2418bbb7-dec4-4aca-907d-42279a6945c6" (UID: "2418bbb7-dec4-4aca-907d-42279a6945c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:55.905220 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.905189 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5" pod "2418bbb7-dec4-4aca-907d-42279a6945c6" (UID: "2418bbb7-dec4-4aca-907d-42279a6945c6"). InnerVolumeSpecName "kube-api-access-qh8m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:55.905327 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:55.905255 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca" pod "2418bbb7-dec4-4aca-907d-42279a6945c6" (UID: "2418bbb7-dec4-4aca-907d-42279a6945c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:27:56.003455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.003432 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-openshift-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:56.003455 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.003454 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh8m5\" (UniqueName: \"kubernetes.io/projected/2418bbb7-dec4-4aca-907d-42279a6945c6-kube-api-access-qh8m5\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:56.003584 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.003465 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2418bbb7-dec4-4aca-907d-42279a6945c6-maas-api-xks-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:56.589662 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.589632 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" Nov 01 23:27:56.589872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.589635 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z9k7m" event={"ID":"2418bbb7-dec4-4aca-907d-42279a6945c6","Type":"ContainerDied","Data":"30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0"} Nov 01 23:27:56.589872 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:27:56.589799 5778 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a333cae30016f6b6005071d98032f5c2aab2b9853e5ccccc5a5a69aa4473d0" Nov 01 23:28:14.390108 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:14.390074 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ztp9b_7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac/manager/0.log" Nov 01 23:28:14.577109 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:14.577076 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fsf74_7c05e5e7-e375-4b2f-93ef-3ba630065e0c/manager/0.log" Nov 01 23:28:15.413124 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:15.413091 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zpxk2_13d1d250-8467-498a-87e7-4037d9f66ee2/proxy/0.log" Nov 01 23:28:16.430337 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:16.430305 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-f4tjk_4993adc2-d160-4b47-b0d3-1d19db148aba/revoke-keys/0.log" Nov 01 23:28:16.933724 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:16.933697 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-z9k7m_2418bbb7-dec4-4aca-907d-42279a6945c6/revoke-keys/0.log" Nov 01 23:28:17.256019 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:17.255935 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lmbk6_a08277ab-2d20-4dc3-b737-5420b2e0425b/revoke-keys/0.log" Nov 01 23:28:17.418403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:17.418369 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c2thl_51378849-677d-4da1-82f1-f2289c386f77/postgres/0.log" Nov 01 23:28:18.772027 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:28:18.771983 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:28:19.012047 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:19.012005 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlfjq_2fec942a-c9c0-424d-826d-5a03f4aa9a5d/manager/0.log" Nov 01 23:28:19.179772 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:19.179750 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7g97m_cd99f908-504c-4f19-b77d-a8add7cd065b/manager/0.log" Nov 01 23:28:19.518643 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:19.518564 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-85s99_2681e5ba-efcc-4beb-bdfd-3cbfe1d30983/manager/0.log" Nov 01 23:28:19.905426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:19.905388 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-62v9s_66266fef-815a-475c-a38f-e5e8de8fa883/manager/0.log" Nov 01 23:28:20.602426 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:20.602395 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5h6rff_4f51c778-961a-495e-9229-2be5253a26bb/istio-proxy/0.log" Nov 01 23:28:21.594621 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:21.594573 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bftkj_e6e8ec16-0274-4d53-8fcf-bc8df04dbed8/istio-proxy/0.log" Nov 01 23:28:22.107412 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:22.107377 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hngcr_72936583-7f29-479e-a06a-d584522d03e0/istio-proxy/0.log" Nov 01 23:28:22.624132 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:22.624104 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qndz9_1080d5b9-3db7-4753-a38c-9293c8c53b63/istio-proxy/0.log" Nov 01 23:28:22.962371 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:22.962340 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp_6da3106a-9735-4255-8a62-a2696577b969/istio-proxy/0.log" Nov 01 23:28:23.492359 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:23.492332 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b8b6447f-8bbxr_a0650f02-2be4-4fad-bd28-2fa18f86c6af/router/0.log" Nov 01 23:28:35.058981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:35.058900 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ztp9b_7f386979-0e3b-4a16-b4e2-a4dc2fe4dbac/manager/0.log" Nov 01 23:28:35.243150 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:35.243114 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fsf74_7c05e5e7-e375-4b2f-93ef-3ba630065e0c/manager/0.log" Nov 01 23:28:36.012112 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:36.012083 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zpxk2_13d1d250-8467-498a-87e7-4037d9f66ee2/proxy/0.log" Nov 01 23:28:36.834694 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:36.834656 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-f4tjk_4993adc2-d160-4b47-b0d3-1d19db148aba/revoke-keys/0.log" Nov 01 23:28:37.341770 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:37.341744 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-z9k7m_2418bbb7-dec4-4aca-907d-42279a6945c6/revoke-keys/0.log" Nov 01 23:28:37.675448 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:37.675419 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lmbk6_a08277ab-2d20-4dc3-b737-5420b2e0425b/revoke-keys/0.log" Nov 01 23:28:37.835318 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:37.835292 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c2thl_51378849-677d-4da1-82f1-f2289c386f77/postgres/0.log" Nov 01 23:28:39.085260 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:39.085229 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlfjq_2fec942a-c9c0-424d-826d-5a03f4aa9a5d/manager/0.log" Nov 01 23:28:39.251675 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:39.251645 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7g97m_cd99f908-504c-4f19-b77d-a8add7cd065b/manager/0.log" Nov 01 23:28:39.585093 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:39.585060 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-85s99_2681e5ba-efcc-4beb-bdfd-3cbfe1d30983/manager/0.log" Nov 01 23:28:39.940813 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:39.940783 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-62v9s_66266fef-815a-475c-a38f-e5e8de8fa883/manager/0.log" Nov 01 23:28:40.455094 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:40.455066 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5h6rff_4f51c778-961a-495e-9229-2be5253a26bb/istio-proxy/0.log" Nov 01 23:28:41.484504 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:41.484473 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bftkj_e6e8ec16-0274-4d53-8fcf-bc8df04dbed8/istio-proxy/0.log" Nov 01 23:28:41.991687 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:41.991659 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hngcr_72936583-7f29-479e-a06a-d584522d03e0/istio-proxy/0.log" Nov 01 23:28:42.474974 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:42.474950 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qndz9_1080d5b9-3db7-4753-a38c-9293c8c53b63/istio-proxy/0.log" Nov 01 23:28:42.800740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:42.800656 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp_6da3106a-9735-4255-8a62-a2696577b969/istio-proxy/0.log" Nov 01 23:28:43.313609 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:43.313577 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b8b6447f-8bbxr_a0650f02-2be4-4fad-bd28-2fa18f86c6af/router/0.log" Nov 01 23:28:48.773490 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:28:48.773448 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:28:51.050189 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:51.050158 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:28:51.120284 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:28:51.120259 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:29:18.773653 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:29:18.773613 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:29:48.772177 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:29:48.772141 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:30:00.142290 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.142202 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:00.144754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.142665 5778 cpu_manager.go:414] "containerMap: removing container" podUID="2418bbb7-dec4-4aca-907d-42279a6945c6" containerName="revoke-keys" Nov 01 23:30:00.144754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.142684 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2418bbb7-dec4-4aca-907d-42279a6945c6" containerName="revoke-keys" Nov 01 23:30:00.144754 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.142832 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="2418bbb7-dec4-4aca-907d-42279a6945c6" containerName="revoke-keys" Nov 01 23:30:00.146179 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.146160 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.149660 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.149628 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-svqdd\"" Nov 01 23:30:00.162264 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.162233 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:00.185393 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.185364 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.185555 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.185483 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntgl\" (UniqueName: \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.287010 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.286982 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.287146 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.287056 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntgl\" (UniqueName: \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.287625 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.287606 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.295588 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.295560 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntgl\" (UniqueName: \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\") pod \"maas-api-key-cleanup-29892930-ss8lh\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.459379 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.459349 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:00.733363 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:00.733335 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:00.735060 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:30:00.735028 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d107853_d25f_45a1_afee_f524d78e2705.slice/crio-f882a7085d0e2f1d058b9a3db8619a0de45c9cb289785358c42a1393b3cc091e WatchSource:0}: Error finding container f882a7085d0e2f1d058b9a3db8619a0de45c9cb289785358c42a1393b3cc091e: Status 404 returned error can't find the container with id f882a7085d0e2f1d058b9a3db8619a0de45c9cb289785358c42a1393b3cc091e Nov 01 23:30:01.307740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:01.307656 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerStarted","Data":"6afdcb9ba9c661fa47b7a1d96c3eb86e857406c1eafa2abd222ba0b96bed5fd4"} Nov 01 23:30:01.307740 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:01.307708 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerStarted","Data":"f882a7085d0e2f1d058b9a3db8619a0de45c9cb289785358c42a1393b3cc091e"} Nov 01 23:30:01.324543 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:01.324495 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" podStartSLOduration=1.324481418 podStartE2EDuration="1.324481418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:01.324040607 +0000 UTC m=+3971.100027438" watchObservedRunningTime="2026-11-01 23:30:01.324481418 +0000 UTC m=+3971.100468227" Nov 01 23:30:02.315107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:02.315070 5778 generic.go:356] "Generic (PLEG): container finished" podID="0d107853-d25f-45a1-afee-f524d78e2705" containerID="6afdcb9ba9c661fa47b7a1d96c3eb86e857406c1eafa2abd222ba0b96bed5fd4" exitCode=60 Nov 01 23:30:02.315519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:02.315135 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerDied","Data":"6afdcb9ba9c661fa47b7a1d96c3eb86e857406c1eafa2abd222ba0b96bed5fd4"} Nov 01 23:30:02.315519 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:02.315360 5778 scope.go:122] "RemoveContainer" containerID="6afdcb9ba9c661fa47b7a1d96c3eb86e857406c1eafa2abd222ba0b96bed5fd4" Nov 01 23:30:03.329353 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:03.329277 5778 generic.go:356] "Generic (PLEG): container finished" podID="0d107853-d25f-45a1-afee-f524d78e2705" containerID="0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803" exitCode=60 Nov 01 23:30:03.329353 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:03.329321 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerDied","Data":"0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803"} Nov 01 23:30:03.329825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:03.329392 5778 scope.go:122] "RemoveContainer" containerID="6afdcb9ba9c661fa47b7a1d96c3eb86e857406c1eafa2abd222ba0b96bed5fd4" Nov 01 23:30:03.329825 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:03.329572 5778 scope.go:122] "RemoveContainer" containerID="0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803" Nov 01 23:30:03.329960 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:30:03.329821 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-ss8lh_odh-ai-gateway-infra(0d107853-d25f-45a1-afee-f524d78e2705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" podUID="0d107853-d25f-45a1-afee-f524d78e2705" Nov 01 23:30:04.336753 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:04.336726 5778 scope.go:122] "RemoveContainer" containerID="0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803" Nov 01 23:30:04.337203 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:30:04.336960 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-ss8lh_odh-ai-gateway-infra(0d107853-d25f-45a1-afee-f524d78e2705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" podUID="0d107853-d25f-45a1-afee-f524d78e2705" Nov 01 23:30:10.564077 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.564002 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vhgqk_9445ebf3-feba-48ea-8eb6-882fd83343ab/global-pull-secret-syncer/0.log" Nov 01 23:30:10.644646 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.644620 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5j2lt_65225fe4-74e2-4b06-9169-7dce163b2fec/connection-checker/0.log" Nov 01 23:30:10.668501 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.668472 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jpxjw_06fbbe23-1f70-4c1e-b82f-5fa72a4166db/connection-checker/0.log" Nov 01 23:30:10.691613 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.691579 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wtzbw_29c2c4f7-79ec-444c-be78-ce43edf6850c/connection-checker/0.log" Nov 01 23:30:10.827087 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.827023 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w5598_62bfdd4e-d511-4e4d-a225-17787716d664/konnectivity-agent/0.log" Nov 01 23:30:10.884068 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:10.884045 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-223.us-east-2.compute.internal_608dfbc50dc552b9f5f883d23954a6ba/haproxy/0.log" Nov 01 23:30:15.366755 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:15.366728 5778 scope.go:122] "RemoveContainer" containerID="0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803" Nov 01 23:30:16.399288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:16.399213 5778 generic.go:356] "Generic (PLEG): container finished" podID="0d107853-d25f-45a1-afee-f524d78e2705" containerID="d5efff30b5b5bfa8ef0376f64ea13b5c0204457f22baa4ccdd1fe9b8f8a4676c" exitCode=60 Nov 01 23:30:16.399288 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:16.399261 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerDied","Data":"d5efff30b5b5bfa8ef0376f64ea13b5c0204457f22baa4ccdd1fe9b8f8a4676c"} Nov 01 23:30:16.399758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:16.399301 5778 scope.go:122] "RemoveContainer" containerID="0d0cbe9ea990381d096b53419419bdd14962387818fa98d220582a5bc59d6803" Nov 01 23:30:16.399758 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:16.399603 5778 scope.go:122] "RemoveContainer" containerID="d5efff30b5b5bfa8ef0376f64ea13b5c0204457f22baa4ccdd1fe9b8f8a4676c" Nov 01 23:30:16.399901 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:30:16.399809 5778 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-ss8lh_odh-ai-gateway-infra(0d107853-d25f-45a1-afee-f524d78e2705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" podUID="0d107853-d25f-45a1-afee-f524d78e2705" Nov 01 23:30:17.426126 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.426094 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:17.597187 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.597155 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:17.733596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.733521 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\") pod \"0d107853-d25f-45a1-afee-f524d78e2705\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " Nov 01 23:30:17.733596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.733585 5778 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\" (UniqueName: \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\") pod \"0d107853-d25f-45a1-afee-f524d78e2705\" (UID: \"0d107853-d25f-45a1-afee-f524d78e2705\") " Nov 01 23:30:17.734038 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.734004 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca" pod "0d107853-d25f-45a1-afee-f524d78e2705" (UID: "0d107853-d25f-45a1-afee-f524d78e2705"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:30:17.736142 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.736113 5778 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl" pod "0d107853-d25f-45a1-afee-f524d78e2705" (UID: "0d107853-d25f-45a1-afee-f524d78e2705"). InnerVolumeSpecName "kube-api-access-qntgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:30:17.835418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.835389 5778 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d107853-d25f-45a1-afee-f524d78e2705-maas-api-service-ca\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:30:17.835418 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:17.835414 5778 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntgl\" (UniqueName: \"kubernetes.io/projected/0d107853-d25f-45a1-afee-f524d78e2705-kube-api-access-qntgl\") on node \"ip-10-0-141-223.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:30:18.001832 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.001755 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlfjq_2fec942a-c9c0-424d-826d-5a03f4aa9a5d/manager/0.log" Nov 01 23:30:18.028215 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.028189 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7g97m_cd99f908-504c-4f19-b77d-a8add7cd065b/manager/0.log" Nov 01 23:30:18.126127 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.126105 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-85s99_2681e5ba-efcc-4beb-bdfd-3cbfe1d30983/manager/0.log" Nov 01 23:30:18.224911 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.224887 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-62v9s_66266fef-815a-475c-a38f-e5e8de8fa883/manager/0.log" Nov 01 23:30:18.413732 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.413700 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" event={"ID":"0d107853-d25f-45a1-afee-f524d78e2705","Type":"ContainerDied","Data":"f882a7085d0e2f1d058b9a3db8619a0de45c9cb289785358c42a1393b3cc091e"} Nov 01 23:30:18.413732 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.413719 5778 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh" Nov 01 23:30:18.413961 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.413752 5778 scope.go:122] "RemoveContainer" containerID="d5efff30b5b5bfa8ef0376f64ea13b5c0204457f22baa4ccdd1fe9b8f8a4676c" Nov 01 23:30:18.449098 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.449077 5778 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:18.459789 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:18.459770 5778 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-ss8lh"] Nov 01 23:30:18.772310 ip-10-0-141-223 kubenswrapper[5778]: E1101 23:30:18.772280 5778 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jgr7" containerName="extractor" Nov 01 23:30:19.835037 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:19.835009 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qsgwz_bc99826a-2d2c-41ae-b55a-e95475056ead/cluster-monitoring-operator/0.log" Nov 01 23:30:20.191659 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.191623 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4d2n_2776ec9b-f80e-438f-a30b-53e86e5b4f2f/node-exporter/0.log" Nov 01 23:30:20.214364 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.214339 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4d2n_2776ec9b-f80e-438f-a30b-53e86e5b4f2f/kube-rbac-proxy/0.log" Nov 01 23:30:20.240036 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.240016 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4d2n_2776ec9b-f80e-438f-a30b-53e86e5b4f2f/init-textfile/0.log" Nov 01 23:30:20.372427 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.372399 5778 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d107853-d25f-45a1-afee-f524d78e2705" path="/var/lib/kubelet/pods/0d107853-d25f-45a1-afee-f524d78e2705/volumes" Nov 01 23:30:20.676362 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676338 5778 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm"] Nov 01 23:30:20.676705 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676692 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676706 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676737 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676742 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676753 5778 cpu_manager.go:414] "containerMap: removing container" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676757 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676758 5778 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676818 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676830 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.676981 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.676837 5778 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d107853-d25f-45a1-afee-f524d78e2705" containerName="cleanup" Nov 01 23:30:20.681107 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.681087 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.687298 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.687275 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"kube-root-ca.crt\"" Nov 01 23:30:20.687452 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.687427 5778 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"openshift-service-ca.crt\"" Nov 01 23:30:20.687607 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.687548 5778 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7bxbb\"/\"default-dockercfg-k9lzk\"" Nov 01 23:30:20.689811 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.689790 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm"] Nov 01 23:30:20.760004 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.759972 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-sys\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.760161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.760090 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-kube-api-access-w4vxh\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.760161 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.760131 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-lib-modules\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.760265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.760204 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-proc\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.760265 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.760252 5778 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-podres\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861458 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861433 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-proc\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861462 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-podres\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861487 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-sys\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861548 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-kube-api-access-w4vxh\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861550 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-proc\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861573 5778 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-lib-modules\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861580 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-podres\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861609 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-sys\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.861837 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.861677 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-lib-modules\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.870105 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.870085 5778 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/2ce8feb5-231f-4357-89a4-7d0cebfa0f85-kube-api-access-w4vxh\") pod \"perf-node-gather-daemonset-x8pfm\" (UID: \"2ce8feb5-231f-4357-89a4-7d0cebfa0f85\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:20.995440 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:20.995370 5778 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:21.274641 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:21.274609 5778 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm"] Nov 01 23:30:21.276529 ip-10-0-141-223 kubenswrapper[5778]: W1101 23:30:21.276502 5778 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ce8feb5_231f_4357_89a4_7d0cebfa0f85.slice/crio-aaee042d200c4ca468cc981f912dd6b89ac7bef7597e8525429b5090bfadcc7f WatchSource:0}: Error finding container aaee042d200c4ca468cc981f912dd6b89ac7bef7597e8525429b5090bfadcc7f: Status 404 returned error can't find the container with id aaee042d200c4ca468cc981f912dd6b89ac7bef7597e8525429b5090bfadcc7f Nov 01 23:30:21.432391 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:21.432361 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" event={"ID":"2ce8feb5-231f-4357-89a4-7d0cebfa0f85","Type":"ContainerStarted","Data":"7b4ae6a8a769db166a4db75cdb0a5b9184e5680780c2cbdf4543cb15e6a77513"} Nov 01 23:30:21.432557 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:21.432409 5778 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" event={"ID":"2ce8feb5-231f-4357-89a4-7d0cebfa0f85","Type":"ContainerStarted","Data":"aaee042d200c4ca468cc981f912dd6b89ac7bef7597e8525429b5090bfadcc7f"} Nov 01 23:30:21.432557 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:21.432510 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:21.454894 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:21.454836 5778 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" podStartSLOduration=1.454822814 podStartE2EDuration="1.454822814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:21.452872984 +0000 UTC m=+3991.228859813" watchObservedRunningTime="2026-11-01 23:30:21.454822814 +0000 UTC m=+3991.230810016" Nov 01 23:30:22.146167 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:22.146137 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2rlrx_1681c152-d4a7-417f-a54e-a7a4b11a2575/networking-console-plugin/0.log" Nov 01 23:30:22.726750 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:22.726721 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/1.log" Nov 01 23:30:22.731596 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:22.731572 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nbf7s_8aa5e010-5d31-46f0-bb5a-a952613af986/console-operator/2.log" Nov 01 23:30:23.840084 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:23.840046 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-znttb_3ad608dd-a3a4-47cb-9702-4cbef303f2f7/volume-data-source-validator/0.log" Nov 01 23:30:25.065405 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:25.065377 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zwrr_010a5ac6-aeef-45ce-8c57-dcebf29427b1/dns/0.log" Nov 01 23:30:25.088476 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:25.088441 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zwrr_010a5ac6-aeef-45ce-8c57-dcebf29427b1/kube-rbac-proxy/0.log" Nov 01 23:30:25.196399 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:25.196378 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5wh4_d2372e60-3b4e-407f-b0ed-6799496e9dd7/dns-node-resolver/0.log" Nov 01 23:30:25.931328 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:25.931297 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xpgg7_41572fd2-3250-4c36-a7be-8c8f733c39c3/node-ca/0.log" Nov 01 23:30:26.952654 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:26.952623 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5h6rff_4f51c778-961a-495e-9229-2be5253a26bb/istio-proxy/0.log" Nov 01 23:30:27.449939 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:27.449909 5778 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-x8pfm" Nov 01 23:30:29.320131 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:29.320080 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bftkj_e6e8ec16-0274-4d53-8fcf-bc8df04dbed8/istio-proxy/0.log" Nov 01 23:30:29.409893 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:29.409862 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hngcr_72936583-7f29-479e-a06a-d584522d03e0/istio-proxy/0.log" Nov 01 23:30:29.525795 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:29.525760 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qndz9_1080d5b9-3db7-4753-a38c-9293c8c53b63/istio-proxy/0.log" Nov 01 23:30:29.586403 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:29.586331 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp_6da3106a-9735-4255-8a62-a2696577b969/istio-proxy/0.log" Nov 01 23:30:29.681790 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:29.681763 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b8b6447f-8bbxr_a0650f02-2be4-4fad-bd28-2fa18f86c6af/router/0.log" Nov 01 23:30:30.604580 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:30.604553 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5h6rff_4f51c778-961a-495e-9229-2be5253a26bb/istio-proxy/0.log" Nov 01 23:30:32.427190 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.427160 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wn5v5_49f57d53-5511-4fc4-86b1-a4884a4d213a/serve-healthcheck-canary/0.log" Nov 01 23:30:32.513877 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.513830 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bftkj_e6e8ec16-0274-4d53-8fcf-bc8df04dbed8/istio-proxy/0.log" Nov 01 23:30:32.597782 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.597755 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-hngcr_72936583-7f29-479e-a06a-d584522d03e0/istio-proxy/0.log" Nov 01 23:30:32.667533 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.667502 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qndz9_1080d5b9-3db7-4753-a38c-9293c8c53b63/istio-proxy/0.log" Nov 01 23:30:32.719518 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.719443 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-d95gp_6da3106a-9735-4255-8a62-a2696577b969/istio-proxy/0.log" Nov 01 23:30:32.783600 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.783570 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b8b6447f-8bbxr_a0650f02-2be4-4fad-bd28-2fa18f86c6af/router/0.log" Nov 01 23:30:32.917173 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.917146 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-nkrwh_d47d3a1e-edf7-4b20-b30c-96449721dbef/insights-operator/0.log" Nov 01 23:30:32.919143 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.919119 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-nkrwh_d47d3a1e-edf7-4b20-b30c-96449721dbef/insights-operator/1.log" Nov 01 23:30:32.938976 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.938957 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jgr7_63988932-7675-4400-beb1-610e2ecfde16/kube-rbac-proxy/0.log" Nov 01 23:30:32.960631 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.960614 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jgr7_63988932-7675-4400-beb1-610e2ecfde16/exporter/0.log" Nov 01 23:30:32.982449 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:32.982400 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jgr7_63988932-7675-4400-beb1-610e2ecfde16/extractor/0.log" Nov 01 23:30:35.469345 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:35.469312 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-dbtw4_ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4/manager/0.log" Nov 01 23:30:35.492058 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:35.492035 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-dbtw4_ebfe1a4a-beb6-41c0-9f3f-9c4e8588b3e4/copy-manifests/0.log" Nov 01 23:30:36.354303 ip-10-0-141-223 kubenswrapper[5778]: I1101 23:30:36.354273 5778 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5h6rff_4f51c778-961a-495e-9229-2be5253a26bb/istio-proxy/0.log"